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1. EXECUTIVE SUMMARY 


1.1 Introduction. Most flight simulators in service today are operated from 
instructor stations where design requirements have been established by sub- 
jective opinion, past experience, and space and equipment constraints. In 
contrast, crew stations of simulators, being replicas of aircraft crew com- 
partments, reflect painstaking, systematic efforts in human engineering and 
pilot evaluation. To improve the overall quality of simulation, then, efforts 
should be directed at improving the efficiency and operability of instructor 
facilities. 


The objective of this study is to develop a method of evaluating the de- 
gree to which an instructor/operator station (10S) design bridges the gap 
between human characteristics and machine requirements. An objective evalua- 
tion methodology should assist the designer in assessing a tentative [0S de- 
sign by identifying devices and functions responsible for poor system perform- 
gnce. A secondary objective of the study was to apply this tool to evaluate 
the effectiveness of various interface layouts and devices. 


The primary purpose of this report is to describe the development, test, 
and application of a computer-assisted evaluation technique which resulted 
from this study. 

1.2 Approach. The I0S is the man-machine system which permits an instructor 
to utilize a simulator and, in particular, its computer in the instruction 
aid evaluation of a trainee. The TOS establishes the compatibility between 
the following: 

(a) The exigencies of training objectives and instructional tasks 

(b) The skills, characteristics and limitations of the instructor 

(c) The functions, characteristics and limitations of the computer. 


The basis for the suitability evaluation of an JOS was defined as the 
time required for an operator to perform an instructional task. This measure 


. is valid only if the task is feasible with the equipment provided. In addi- 
tion, the measure is meaningful only if the instructor performs the task cor~ 
rectly. 


These objectives were realized through the following steps: 
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(a) Delineation of the instructional tasks and activities. 
(b) Consideration of man-machine interface factors. 


(c) Construction of a typical sequence of instructional tasks ur a re- 
presentative task profile (RTP). 


(d) Definition of typical instructor tasks or activities and a break- 
down of those tasks in terms of units of time required for comple- 
tion. 
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(e) Construction of a process to yield a quantitative evaluation of an 
IOS design. 


(f) Validation of that process through comparison with empirical data. 


(g) Demonstration of the utility and practicality of the evaluation 
method by the general user. ae 
In carrying out these steps, building blocks were’ dévloped including an 
Instructional Task List, Representative Task Profile, and Typical Activity 
Modules. These were compatible with SAINT, the computer analysis package 
used. 


1.2.1 Instructional Task List. A list of the activities required to prepare 
and conduct a training flight was developed from lesson plans prepared for 
the United States, West German, and Canadian Armed Forces. The resulting 
Instructional Task List (ITL) provides the designer/evaluator with a list of 
Zz these typical tasks. Any mission can be constructed by selecting a suitable 
E : sequence from the ITL (a typical task is ARRANGE MAPPING DISPLAYS). 


1.2.2 Representative Task Profile. A typical training mission, including a 
wide variety of instructional tasks and assembled with a reasonable sequence 
of items selected from the ITL, was defined as an RTP. This RTP definition 
Bes was useful in evaluating the suitability of alternate I0S designs to the 
Be tasks the instructor must perform in a training exercise. Instructive activ- 
z ities (such as the assessment of trainee performance) were included in the 
a ITL and, hence, in the RTP since they are dependent on information presented 
Bi to the instructor by the I0S. Thus, the method of their presentation was one 
: factor to be evaluated. 


a 1.2.3 Typical Activity Modules. Through analysis of the RTP, repetitive se- 
2 quences of tasks could be identified and grouped. For example, the RTP task 
ARRANGE MAPPING DISPLAYS was divided into groups of tasks such as read, dis- 
ea These groups of tasks were defined as typical activity modules 
TAMs). 


1.2.4 SAINT. An existing network and modelling program, SAINT (systems an- 
alysis of integrated network of tasks), was selected as a ready-made and con- 
venient tool to form the model of the I0S interface. SAINT was developed at 
the Aerospace Medical Research Laboratory, Wright-Patterson AFB, Ohio, to 
model, in network form, sets of tasks performed during the course of a mis- 
sion. SAINT obtains mission performance measures for networks which repre- 
sent a mission consisting of a set of tasks performed by a crew of operators. 


A computer simulation approach is used to obtain the performance mea- 
sures. Human engineering considerations are included through parameters asso~ ° 3 
: ciated with tasks, precedence relations between tasks, and factors affecting 4 
i. crew performance. The network simulates the events of the TAMs and generates 
time performance data. These data can then be used to model the tasks of the 
RTP, predicting overall performance for segments of the training mission. The 
‘model is completely adjustable through parameter sets defining tasks. The 
network structure is such that interactive effects between tasks and workload 


ae “. 
- Re a ee 2 ek cana inl aetna cae nant ml eet ta ne 


are evaluated by iteration of the model and not be the arbitrary inputs of 
the user. 


By this approach, and due to the use of an active computer model, two 
phases of evaluation became available. The data obtained from the TAMs yield 
information as to the,suitability of a device to have a task performed on 
it. The data obtainea from SAINT, when modelling the RTP, yield informa- 
tion as to the suitability of the entire 10S (of which the interface is a 
part) to the running of a flight training mission. 


1.3 Evaluation Methodology. For this study to be effective, an evaluation 
methodology was required which did not rely on a high level of programming 

expertise on the part of the user. To achieve this goal, each part of the 

descriptive chain formed by the RTP and TAMs was implemented in a standard- 
ized format. 


The RTP was formed as a standard training mission. Running the RTP with 
SAINT with the parameters chosen for the I0S in question yields a quantita- 
tive description of the I0S. The TAMs have been structured to include as 
many as possible of the actions normally performed by an instructor. The 
user chooses the actions applicable to the IOS under evaluation. The TAMs 
are supplied to the user in standardized form. SAINT provides the ability to 
designate tasks in a network as dormant by defining the time required to per- 
form the task as zero. Thus, the user need only choose an appropriate TAM, 
designate the unnecessary tasks as dormant, and choose from the supplied in- 
dex of parameter sets the parameters which best describe the tasks as per- 
formed on the candidate 10S. These data can easily be inserted into the data 
files supplied for each TAM. The TAMs can now be run by SAINT. 


The data supplied by the TAMs are then used to evaluate instructor tasks 
within the RTP. As with the TAMs, the RTP is supplied to the user fully 
constructed and with its data file. 


The user is supplied with standard forms for each TAM and each portion 
of the RTP mission. The user need only choose the appropriate parameter set 
(from the parameter set index) which best describes the interface in ques- 
tion, and run the descriptive networks witn those data to obtain a detailed 
analysis of the IOS design being evaluated. 


1.4 Conclusions. The evaluation methodology proved to be an efficient and 
easy to use procedure with which to evaluate the design of an IOS, SAINT 
displayed a very high sensitivity to smal] variations in the data, and there- 
fore to small differences in I0S design. 


The descriptive chain, made up of the RTP, ITD (instructor task descrip- 
tion) and TAMs, was arranged in a checklist form by which the user could 
easily choose those characteristics pertinent to the I0S being evaluated or 
designed. The parameters sets necessary to supply the times required for 
each instructor set could easily be inserted into the supplied data files for 
each portion of the RTP and each TAM. 
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1.4 Conclusions (Cont'd) 


The model was validated by choosing a particular task in the RT? and 
varying the parameters associated with it (i.e., time required for the task, 
operator speed, accuracy, and stress threshold). The results of these varia- 
tions were reflected in the number of successful missions, times for comple- 
tion of the missions, and average stress levels on the operators, The model, 


and’ the SAINT program, proved to be very sensitive to these data perturba- 
tions. 


This validated methcclology was then used to evaluate two designs of the 
10S for a typical jet fighter trainer. Both designs contained the same 
hariware, but the hardware configuration differed. By adjusting the model of 
the 2TP ¢o fit both the configu: ,i ions (#1 and #2) and running the data on 
SAINT, it was determined which configuration was more advantageous to the 
running ef a training mission. 


The methodology was then used to evaluate three candidate interface 
techniques. It was here that the sensitivity of the model to data perturba- 
tions vas rigorously tasted. A task was designated to be performed on a par- 
ticular interface device. When different interface devices were implemented 
cn an IOS, although these deviccs may be ised 43 infrequently as twice per 
leg of the RTP, different numbers of scuscessful missions, different times of 
compte. ton and different stress levels were reported by the software. 


To summarize, the cey.loned methodology proved to be easy to implement, 


highly sensitive, and prec'se. As more ergonomic data are found to: quantify 


the fustructor activities, the model will become more accurate. In general, 


the model can be used to select components, evaluate layout and indicate the 
acceptability of [0S designs. 


Qn 


aes eee pede See Nee ee fer 


2. INTRODUCTION 


2.1 Study Objectives. Technology exists today whereby training objectives 
can be achieved through automatic training and performance evaluation. Auto- 
mation has obvious advantages, sucii as uniformity of training and elimination 
of instructor bias; however, most investigators and users find that positive 
instructor control and involvement in a training exercise are essential fac- 
tors of objective and complete training. Hence, the main goal of this study 
was to establish a systematic process to aid in the evaluation and design of 
instructor/operator stations. This goal introduces the need for a systematic 
division of tasks between the instructor and the man-machine interface. 


2.1.1 Interface. The main responsibility for a given activity may be as- 
signed to either the man or the machine, but the training objectives and re- 
sults in both cases remain the same. Only the levels and modalities of the 
interface change as the I0S designer attempts to balance functional flexibil- 
ity and freedom of control against machine support to attain the desired sys- 
tem performance at an acceptable instructor workload. The choice and arrange- 
ment of controls, together with the associated software and hardware func- 
tions, comprise the IOS man-machine interface. 


2.2 Approach to Meeting Study Objectives. The study first required a systeni- 
atic review and delineation of instructional tasks (e.g., insert lesson plan, 
set up scenario) and an explanation of the factors intrusive on their per- 
formance. 


The establishment of an objective method to evaluate the suitability of 
the interface to typical IOS tasks in quantitative terms was the second step 
in the approach. 


The final step was the development of a means to predict man-machine 
system performance of a given IOS design for a profile of representative 
tasks, 


2.2.1 Nature of Instructor Tasks. Based on experience with flight simula- 
tors and lesson plans prepared for the United States, West German, and Cana- 
dian Armed Forces, a list of activities required to prepare and canduct a 
training flight was developed. This ITL, Figure 2-1, provided a systematic 
breakdown of the overall training objectives into their component steps. 


2.2.1.1 Pacing. In preparing the ITL, potential pacing sources for each of 
the tasks were identified as follows: 


Instructor: free choice as to time of activity 


Simulation: timing determined by the progress of the simulated 
flight 


Trainee: activity demanded by particular needs of the trainee 
based on trainee's long term training status and performance 
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Figure 2-1 Typical ITL -Page 


(the entire ITL 
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Equipment: method of command input determined basic task time 


(e) Lesson: timing determined by the unit sequence to completion of a 
lesson 


(f) Indirect simulation: summary interpretation, monitoring activity 
dictates timing 


(g) Maintenance: requisites of maintenance program determine timing 
(as in liaison with maintenance personnel). 


2.2.1.2 Tasks. A typical training mission representing a wide variety of 
instructional tasks was assembled with a reasonable sequence of items sel- 
ected from the ITL. This Representative Task Profile (RTP), Figure 2-2, is a 
sequence of instructor activities performed on an I0S (e.g., ACTIVATE MAL- 
FUNCTION). This breakdown was to evaluate the Suitability of an I0S design 
to the activities or tasks the instructor performed in a training mission. 


2.2.1.2.1 Task Building Blocks. Assuming a conventional complement of I0S 
displays and controls, the tasks of the RTP could be broken down into basic 
building blocks as follows: 


a) Spotting the change in displays 


b) Evaluating the significance of that change 


d 


(a) 

(b) 

(c) Deciding on the required response 
(d) Executing an input if required 
(e) 


e) Monitoring the resulting machine response. 

This breakdown of the RTP tasks resulted in a set of 19 descriptors 
which cover all the tasks in the RTP. These descriptors (e.g., ASSESS CREW 

3 COOPERATION, DISPLAY MANAGEMENT) comprise the title components of the Instruc- 
di tor Task Description (ITD) List (Appendix C) (see Figure 2-3). 


2.2.1.2.2 Task Categories. Upon examination of the items within the ITO, it 
was found that three basic categories of tasks exist. These are as follows: 


(a) Operation Tasks 


(1) Activating manual controls or otherwise making inputs to eli- 
cit simulator functions necessary to conduct the training 


(2) Participating in the simulation by making inputs representing 
the outputs of external sources (e.g., radio messages) 


(3) Effecting changes such as deviating from a preprogrammed se- 
quence or set of simulated conditions. 
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is available in Appendix C 


~3 Typical ITD Page 


Figure 2 


(the entire ITD listing 
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(b) Instruction Tasks 


(1) Monitoring trainee performance an? techniques 


(2) Formal assessment of trainee proficiency and performance 
(3) Coaching, remedial comments, and demonstrations. 
(c) Documentation Tasks 
(1) Recording data for performance evaluation 
(2) Making written notes. 


This categorization aided in determining which particular aspects of 
the IOS could be evaluated. All operational tasks and the documentation task 
of recording data are subject to evaluation because they can be performed on 
a variety of interface hardware. The location of the interface devices, as 
well as the type of interface device, can atfect the efficiency with which a 
task may be performed. The instructional task of monitoring trainee perform- 
ance can also be evaluated because the ability to monitor performance depends 
on the manner in which this information is presented to the instructor (i.e., 
interface methods). 


Not only were the tasks listed, but the step by step procedures re- 
quired to perform a task were incorporated in the ITD. These elements of the 
ITD were described, in terms of how they were performed, in the IOS Activities 
List (also included in Appendix C) (refer to Figure 2-3). For example, an 
ITD item may read "Verify against standard procedure." The I0S activity will 
describe how that is done, in this case, "Read procedure checklist." 


The procedures set forth in the IOS Activity List are further re- 
solved into the step by step actions required to carry out a task on a par- 
ticular interface device. These procedures (Figure 2-4) were arranged into 
Typical Activity Modules (TAMs). The TAMs were written to describe the per- 
formance of operation tasks on- various interface devices. A TAM describes a 
typical instructor/IOS activity in detail sufficient to match most task re- 
quirements to most I0S configurations by simply selecting the appropriate 
TAM, thus avoiding need for individual analysis. This was the final step in 
the resolution of instructor tasks. 


2.2.2 Analysis by Subsystems. Instructor activities involve many physical 
and subjective factors and conditions. Success or failure in achieving de- 
sired system performance depends on -complex interactions between the skills 
and limitations inherent in the human operator, the capabilities and func- 
tions of machine elements toward training problem presentation, trainee eval- 
uation and instructor facilities, and the requirements of instructional task 
activities, as in representing the required sequence of events, pacing func- 
tions and trainee evaluation results. This grouping suggests three inter- 
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Figure 2-4 Typical Page from the TAM Library 
(the entire TAM listing is available in Appendix F) 
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connected, but essentially dependent categories or subsystems. 


When these interfaces are compatible, the resulting continuous system 
will be amenable to analysis in terms of individual subsystem performance 
without detailed knowledge of the processes internal to the subsystems. 


2.2.2.1 Man and Machine Subsystems. The human is inherently incompatible 
with the machine because speed and language differences generate gaps between 
the human sensory and neuromotor skills and the machine input/output capa- 
bilities. It is the I0S interface methodology which must provide appropriate 
matching functions and establish continuity and compatibility. In this con- 
text, the I0S design parameters could be definéd in terms of these matching 
functions, and evaluation would consist of testing the available I0S func- 
tions against them. As discussed later, this capability certainly exists in 
the evaluation methodology developed in this study, but a mature data base is 
required before reliable and repeatable results can be expected for the many 
task/interface combinations possible. Meanwhile, the reverse process is ap- 
plicable, whereby various 10S devices can be selected and tested for suit- 
ability against typical or selected tasks, the entire 10S design and man-mach- 
ine performance can be assessed against a representative task profile. 


The principal contributions of the human, as applicable to. this ana- 
lysis, are: 


(a) Sensory skills and perception 

(b) Cognitive skills, pattern recognition, long- and short-term memory 
(c) Decision making ability, assessment of intangibles 

(d) Neuromuscular skills, complex motor performance. 


Compared with machine functions, these represent highly complex cap- 
abilities in areas such as pattern recognition, as well as limitations in 
others, for example, low reliability of short-term memory and relatively low 
speed in most functions. Furthermore, human performance may be significantly 
altered by factors related to dynamic task conditions, such as workload and 
fatigue. Since these influences may assist, as well as inhibit, human: per- 
formance and since they may be viewed as products of the task subsystem, we 
decided to declare the basic human contributions constant, but vary the "cost 
of performance" with the overall task profile. This stipulation permits the 
application of ergonomic performance data to estimate the time required to do 
elemental tasks using a selected IOS interface method or device. The actual 
time is then the product of the basic time plus factors derived from dynamic 
task conditions. 


2.2.2.2 Task Subsystem. The main output of the task subsystem is the se- 
quencing and pacing of the events required to achieve the training objec- 
tives. The pacing of activities has a decisive impact on the scheme of man- 
machine task allocation and therefore on the 10S interface methodology. For 
the present purposes, instructional tasks and training objectives are .consid- 
ered fixed and unalterable. The RTP may be performed by using different I0S 
devices and functions, but omission of any task constitutes a failure to at- 
tain required system performance. 


2.2.3 Use of Computer Based Model/SAINT. A number of pencil-and-paper me- 
thods based on elemental logic, statistics, and queuing theory are available 
to analyze the interaction of tasks and activities. Link diagrams establish 
the strength of correlation between two work stations and thereby the neces- 
sity to walk from one to the other.’ The Petrie Network method consists 

of two types of elements: places and transitions, connected by directed ed- 
ges or arcs.” Tokens are used to indicate that all conditions (places) 

have been fulfilled to fire a transition (start an activity). The execution 
of a task uses up the tokens. Que to the complexity of the tasks in IOS 
evaluation, it became evident during the preliminary study phase that a more 
powerful tool would be needed and that the SAINT computer model developed by 
the U.S.A.F. Aerospace Medical Research Laboratory was eminently suitable for 
the purpose. 


SAINT is a combined network modelling and simulator technique. SAINT 
was designed to model, in network form, sets of tasks performed during the 
course of a mission. A SAINT network consists of tasks which, when complet- 
ed, can satisfy precedence requirements for other tasks. Operators flow 
throughout the network from task to task performing those tasks. SAINT tasks 
have input sides, descriptors and output sides associated with them. 


2.2.3.1 SAINT Task Input. The input side of a task contains the number of 
tee aa tasks which must be completed before the task can be begun (re- 
leased). 


2.2.3.2 Task Descriptors. The task descriptors are the heart of the net- 
work. The time required to perform a task is characterized by a distribution 
type and a parameter set. 


1 


MISUNAS, 0, Petri nets and speed independent design. ‘Communications 
of the ACM, August 1973, Vol. 16, No. 8. 


PRITSKER, A., et al. SAINT: Systems Analysis of Integrated Network 
of Tasks. Aerospace Medical Research Laboratory, Wright-Patterson Air 
Force Base, Ohio, 1974. 
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2.2.3.2.1 Distribution Types. Sampling can be performed from any of the 
following distribution types: 


(a) Constant 
Normal 
Uniform 
Er-lang 


Log normal 


(b) 
(c) 
(d) 
(e) 
(f) Poisson 
(g) Beta 
(h) Gamma 
(i) Triangular 
(j) Beta fitted to three estimates 

) Constant divided by a scale factor. 
Parameter sets contain the mean, minimum and maximum times, and in- 
clude the standard deviation associated with the distribution. Parameter 
sets are identified by a parameter set number. Operator characteristics also 
have an effect on task duration and these are discussed in paragraph 2.2.3.4. 
2.2.3.2.2 Task Types. There are six possible task types in a SAINT network: 

a) Single operator: task performed by only one operator 


( 

(b) Joint operator: task perforiaed by more than one operator 

(c) Equipment: simulates a piece of equipment and has no operator 
( 


d) Cyclic: task used to provide a delay time until a following task 
can start 


(e) Either: task performed by either one of a set of specified opera- 
tors 


(f) Gap filler: task simulating an activity performed periodically 
when time allows. 


2.2.3.2.3 Task Essentiality. Each task perforined by an operator has a 
graded -essentiality associated with it. Task essentiality is graded on a 
scale from zero to oné, with one as essential. The effect of this essential- 
ity is to permit the skipping of low essentiality tasks when the amount of 


essential time remaining to the mission is insufficient to complete the net- 
work. Essential time is an estimate of that time required to perform the 
essential tasks remaining to the mission. 


2.2.3.3 SAINT Output. The output side of a SAINT task represents a branch- 
ing or decision operation. Following completion of a task, a selection is 
made as to which branches emanating from the task should be activated. 
SAINT's five branching types are: 


(a) Deterministic: all branches emanating from the task are activated 


(b) Probabilistic: each branch emanating from the task has an asso- 
ciated probability of being selected. Only one of the branches is 
selected, based on a number drawn from a zero-one distribution 


{c) Modified probabilistic: this is the same as probabilistic branch- 
ing except that the probabilities associated with the branches 
change by a specified amount each time the task is released 


(d) Conditional "take first": the branches are ordered and éach branch 
has a condition associated with it. The branch whose condition is 
met first is activated 


(e) Conditional "take all": this is the same as conditional “take 
first" with the exception that all wranches where conditions are 
met are activated. 


2.2.3.4 Operator Characteristics. Operator characteristics are used in con- 
junction with the task oriented concepts to inake a mission operator-specific, 
and modify the time required to perform a task, as follows: 


(a) Speed: this factor allows for the simulation of operators who are 
faster or slower than an average operator. 


(b) Accuracy: this factor allows for the simulation of operators who 
may be more accurate or less accurate than the average operator. 


(c) Stress threshold: this factor is the value which, when exceeded, 
causes the operators' performance to degrade. The workload stress 
on an operator is the time pressure imposed by a discrepancy between 
the amount of work to be done and the time allowed to do it. 


(d) Goal gradient: this factor introduces an increase in performance 
relative to the proximity of a goal or end point. 


2.2.3.5 SAINT Analysis. SAINT performs its analysis in two parts: bench- 
mark iteration and actual mission performance calculations. The benchmark 
runs obtain estimates of the time required for the operator to perform each 
task and, from this information, calculate the essential time remaining, the 
nonessential time, and the idle time. None of the operator characteristics 
outlined are used in benchmark calculations. 
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In the actual mission performance runs, SAINT provides estimates of 
EB . the probability of successfully completing the mission, the times of comple- 
tion, and the minimum, maximum and average stresses incurred. 


2.2.3.6 Static Analysis. - TAMs. Through SAINT's capabilities, every rea‘lis- 


a tic approach to evaluation for design can be explored. The static division 
oh of elemental activities consists of the TAMs. TAMs model isolated functions 
= with no regard to a lesson plan context, surrounding activities, or other 

BS] TAMs. TAMs, by title, define an !0S activity (i.c., the utilization of I0S 


hardware) and describe the procedures necessary to carry out the activity. 
These procedures are the basic human functions required in the use 

F . of a device (i.e., display, keyboard, switch, etc.). The basic functions are 

ae as. follows: 

(a 

(b 


Visual Access 
Manual Access 
(d) Reads 


) 
) 

(c) Activations 
) Decisions. 


TAMs also contain basic functions of the hardware of the J0S. 
These functions are: 


(f) Attention-getting 
(g) Display delays 


(h) Information presentation task (IPTASK) characteristics (i.e., in- 
tensity of display, contrast in display, clutter, etc.). 


run by SAINT, comprise the static phase of the evaluation (i.e., the isolated : 


EF | 
Hi The interaction of the human and hardware factors, when a TAM is | 
performance of an isolated IOS activity on an interface device). | 


incorporates the static phase into a representative lesson plan as an inte- 

gral part of the evaluation. The data obtained from running the TAM through 
SAINT (i.e., average time of mission completion and probability of success) 

are inserted into the RTP model. The TAMS now interact with other TAMs in 

this representative lesson plan. From this dynamic model, the effects of all 
aspects of the I0S, in conjunction with TAMs and the application of the I0S 4 
to the lesson plan, are set forth. The results of a TAM describing a parti- 1 ¥ 
cular device may indicate that, in itself, that device is more suitable than aS 
several others. However, when that TAM is placed into the main model, its ne. 
Suitability to the particular IOS may be less than desirable. For example, a a 
dedicated push button to call up a CRT page is. faster than a keyboard inser- 


2.2.3.7 SAINT Dynamic Evaluation. The dynamic aspect of evaluation/design 


16 


tion for the same page. Insertion of the push button TAM into the dynamic 
model, however, may indicate that because of the location of the push button 
and the number of times it may be accessed, it is not the more efficient 
device for that function on that [0S. 


2.3 Summary. Figure 2-5 presents a graphic representation of the study ap- 
proach. An instructional task list was compiled. ITL items were arranged to 
form a typical lesson plan (RTP). The tasks of that RTP were then subdivided 
into basic building blocks (e.g., READ, DISPLAY, EVALUATE) in the ITD. The 
actual method of performing the instructor tasks was expanded into the I0S 
Activities List. It is here that the distinction between operator-only tasks 
and machine-assisted tasks surfaces. The tasks performed on the IOS are des- 
cribed in terms of their most basic functions on a particular interface de- 
vice in TAMs. Ergonomic data, quantifying the time durations to perform 
these basic functions, are assigned to the TAMs and are analyzed by SAINT to 
supply the static phase of the evaluation. The data obtained from the static 
phase are then used as input to the RTP model. SAINT's anatysis of these data 
provides the user with the model performance data. 


2.4 Document Structure. The main body of this study consists of four sec- 
tions. Section 2 presents an overview of study objectives. Section 3 de- 
tails the development of the objective evaluation methodology used in analyz- 
ing and evaluating the designs of i9Ss. Candidate interface techniques, the 
approach to the analysis, the actual development of the evaluative model and 
the validation of this model are all dealt with in Section 3. 


Section 4 applies the developed methodology to the evaluation of two 
off-board 10Ss and to three candidate interface techniques. 


Section 5 summarizes the model effectiveness and the benefits to the 
user and includes recommendations for further work. 
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Figure 2-5 Evaluation Methodology Development (Graphically) 
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3. DEVELOPMENT OF OBJECTIVE EVALUATION METHODOLOGY 


3.1 Candidate I0S Techniques 


3.1.1 Introduction, The efficiency of the 10S determines the instructor/ 
operator workload and affects the success probability of a training session. 
Each interface technique or device should be designed for optimal man-machine 
interaction for specific tasks. The proper blend of techniques yields an 
efficient 10S. Some of the more common interface techniques are presented in 
this section, as well as some lesser known new technology devices. A brief 
description of devices operations is given and their advantages or disadvan- 
tages are exposed. 


A survey of CAE simulators provided an initial list of candidate tech- 
niques which was extended by reviewing additional literature. Assessment of 
the applicability of a device to a specific task or group of tasks stems from 
a general knowledge of simulators and training, a data gathering trip (Appen- 


dix H), and other reference literature, particularly AFHRL-TR-77-10 and 
AFHRL-TR-77~-50. 


3.1.2 General and Specific Purpose Keyboards. The general purpose keyboard 
is a commercially available unit with an arrangement of keys in a standard 
typewriter configuration with no backlighting. Special purpose keyboards are 
available with optional backlighting. However, these tend to be costly and, 
in general, only simple matrix configurations can be purchased. 


For the purpose of modelling, both the standard typewriter arrangement 
and a special purpose keyboard are considered together. These are input de- 
vices, usually operating in conjunction with a CRT (which provides feedback). 
They are used for tasks requiring alphanumeric input, examples of which might 
be selecting a new CRT display format, selecting a particular item on a CRT 
for input, or specifying a new value for a system parameter. 


Unless a requirement exists for inputing alphabetics, a special pur- 
pose keyboard composed of numerics and a limited number of special purpose 
backlit keys is preferred. 


4 
4 
a 
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3.1.3 Touch Panel. Used in conjunction with 2 CRT (which provides feedback), 
this input device provides touch sensitive areas which are activated when a 


finger or other object interrupts one of several light paths. The applica- 

tion of this device is limited by low resolution, and it may be difficult to y 
use if the I0S is based on a motion platform; also, the operator must be 4 
within arm's length to actuate the device. 4 


Two advantages of touch panels are that the touch sensitive areas are 
programmable, making it possible for one touch area to take on several func- 
tions, and there are no moving parts as in the case of switches or buttons. 
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3.1.4 Light Pen. This input device is best suited for tasks which require 
specifying a point on a CRT display, such as selecting an item from a menu. 
It is hand-held and connected to the computer or display controller interface 
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by a flexible cable. When not in use, it is held hy a clamp near the side or 
front of the CRT. A switch on the light pen opens a shutter allowing the 
light from the CRT to generate a signal which is then sent to the computer. 


Its limitations are that it is useful only in proximity of the CRT and. 
would be difficult to use on a motion-based I0S. 


3.1.5 Voice Input/Output. The voice input/output devices are the essential 
components of the overall aircraft simulation through which the instructor/ 
operator fills the role of external voice sources and receivers (e@.g., con- 
trol tower, another aircraft crew). These same channels are used for nonsimu- 
lation exchanges (e.g., crew briefing). 


Voice messages are transmitted and received over a headset connected 
through a jack at the I0S or a microphone/loudspeaker combination. In both 
cases, all crew transmissions are always available at the headset, earphones 
or loudspeaker, but activation of a foot switch, knee bar or HOT MIC selec- 
tion is required for instructor/operator messages to be transmitted to the 
crew. Access to these devices is limited, especially for the microphone/ 
loudspeaker comb nation. 


3.1.6 Tactile Displays. These devices provide tactile prompting (e.g., a 
switch that puts pressure on a resting finger when action is demanded). These 
devices have limited applications at an 10S since the relevant parts of the 
body must be in contact with the device at the appropriate instant to be 
effective. 


3.1.7 Foot Pedal and Kneebar. The foot pedal is a foot-operated switch nor- 
mally used as a press-to-talk switch for the instructor/operator. The foot 
pedal is connected to the IOS by a certain length of cable and to use the de- 
vice, the instructor/operator must be within reach of the pedal. 


A kneebar is a knee-activated switch under the console and is normally 
used for the same purpose as the foot peda}; however, it is intended for use 


3.1.8 CRT Display Cursor Contro) Devices. This category includes devices 
such as track balls and joysticks in conjunction with a CRT. One defines a 
positicn on the CRT screen by moving a displayed cursor to the desired posi- 
tion. In the absence of an input command the cursor maintains its last posi- 
tion, 


Track ball operation involves rotating the exposed portion of a sphere a 
in the desired direction, with the cursor following. A button mounted on the '. 
trackball unit is used to initiate ap action. Joystick operation produces ie 
the same result by swivelling a secured short stick and also operating a but- 
ton to initiate an action. | 


There are many other devices or means of positioning a cursor on the a: 
CRT; however, these two are the most popular. The use of two controls, one : 
for X and one for Y cursor position, should be avoided since the positions 
are difficult to coordinate. 
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3.1.9 Dedicated Push Buttons. One of the most popular man-machine interface 
devices is the dedicated function push button. It is used primarily for con- 
trolling on/off type functions where alternate selections of the push button 
switch the function between its two states. One of the best configurations 
is a push button which uses two colors to indicate its alternate states. In 
this way, a burned out lamp can be found quickly (switch unlit) as opposed to 
a configuration where the unlit condition is actually one of the valid push 
button states. 


Within the I0S push buttons may be isolated or grouped in clusters. 
Caution must be taken to avoid large groups of buttons. 


3.1.10 Speech Recognition/Synthesis. Devices to automaticaliy generate or 
recognize speech are an example of new techniques which are being incorpor- 
ated into flight simulator instructor/operator stations today. Speech recog- 
nition systems allow a computer to react to human voice commands. Speech syn- 
thesizers are devices driven by a computer to generate near-human words or 
phrases. 


In both directions (recognition/synthesis), the machine vocabulary is 
limited and exact. The disadvantage of incorporating such systems is that in 
most systems, when talking to the machine, deliberate silent periods must be 
inserted between words and the recognition system must be programmed to re- 
cognize a specific person's voice. 


3.1.11 Summary. This section introduces some of the common and new inter- 
face techniques which have been, or could be, implemented at an 10S. Where 
possible, the applicability, suitability and advantages of the devices have 
been discussed. 


Three of these interface techniques (dedicated push buttons, light 
pen, and keyboard) were chosen as candidate interface techniques and the jiin- 
pact of implementing them on an IOS is examined fully in Section 4. 


3.2 Analytical Approach 


3.2.1 Time as Principal Criterion. The activities taking place at the 10S 
are exchanges of information between complex subsystems (i.e., human, machine 
and task) across an interface. This concept presumes the interface to be 
continuous, that all necessary and pertinent information is available to the 
Operator in the appropriate format at any time in the training session. In 
return, the operator must decide on the nature and timing of inputs te be 
made to achieve instructional objectives by appropriate machine activity, and 
to deliver the necessary commands to the machine subsystem by means of suit- 
able control devices. However, each of the interface devices treated in Sec- 
tion 3.1 is capable of effectively enabling the achievement of these goals. 
The logical, task-oriented placement of several of these devices is more ef- 
fective than employing these interface techniques in a complex conglomera- 
tion on an I0S. An IOS extremely well suited for basic flight training may 
be totally unfit (although all necessary instrumentation is present) for ad- 
vanced tactics training. The reason for this suitability or lack of suit- 
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3.2.1 Time as Principal Criterion (Cont'd) 


ability is the workload stress imposed upon the operator. 


Simply stated, workload stress is the time pressure imposed on an 
Operator by a discrepancy between the amount of work to be done and the time 
remaining in which to do it.’ This time pressure depends on parameters 
such as the task complexity and the importance of the task. Stress affects 
operator performance. Thus, an operator may be able to conduct easily a 
flight training mission on an IOS, but in conducting an advanced tactics 
training mission on the same I0S the operator may be confronted with an 
arrangement of interface devices which needs more time to operate than the 
operator can give. 


An axiom developed for the analysis of spacecraft crew workloads bas 
been adapted to suit the present purposes: 


A qualified person, that is a properly trained 
and prepared simulator instructor or operator, 
can, and will achieve, error-free performance 
in any 10S task, using almost any interface 
equipment, provided he/she has unlimited 

time. In all other cases, his/her performance 
and the probability of achieving the object- 
ives defined by the training requirements wil] 
be a function of his/her total (physical and 
mental) workload arising from the instruction- 
al, operative, and menitoring tasks assigned 
to him/her and from the physical and temporal 
relationships applicable during the perform- 
ance of these tasks. 


Thus, time and the effects of excess time or insufficient time were 
the principal criteria in this development of the evaluation methodology. 


Whether or not optimal or even essential compatibility exists at the 
10S is by no means obvious from its physical appearance or even from a cur- 
sory ergonomic analysis. The designer/evaluator needs a tool to systematic- 
ally assess the suitability of [0S equipment and functional features for each 
typical task that may be assigned to the instructor, and to test this suit- 
ability in the context of the activities of a representative task profile. 
The tool is the evaluative model which, through the use of SAINT, analyzes 
mission performance in terms of time, and the factors which time affects 
(e.g., stress, boredom). 


Pritsker, A., et al. SAINT: Systems Analysis of Integrated Network 
of Tasks. Aerospace Medical Research Laboratory, Wright-Patterson Air 
Force Base, Ohio, 1974. 
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This approach permits the analysis and comparison of all functional 
aspects and design details of the i0S as generators or consumers of time, or 
more precisely, uncommitted operator workload capacity. The "cost" of insuf- 
ficient information or inadequate machine support appears as excessive time 
required to complete a given task since the operator has to perform 
additiona} work to establish the necessary conditions and satisfactory system 
performance. 


3.2.2 Man-Machine Interfacing Consideration. The considerations of man-mach- 
ine compatibility and performance are almost endless. Most I0S activities, 
however, may be grouped accurately into the following five categories: 


(a) Perception: becoming aware of new information 
(b) Recognition: evaluating, understanding or decoding iaformation 


(c) Decision making: intellectual activity leading to a change in be- 
havior 


(d) Action: physical activity leading to a command input to the machine 


(e) Monitoring: a combination of perception and recognition to ascer- 
tain that the desired activity is being obtained. 


The effects the interface devices, or the layout of the interface 
within an IOS, have on the operator is of great importance in evaluating the 
design of an 10S. For example, if a device intended to attract operator at- 
tention is situated in a field of other similar devices, there is a strong 
possibility that this attention-getter will go unperceived by the instructor. 
In another case, information may be presented to the instructor in a format 
which may not be recognizable or useful. The former is an example of a poor- 
ly designed layout while the latter is a poor choice of interfacing device; 
both affect the performance of a mission. It will be shown later that the 
modular network structure of the evaluative model provides for the represent- 
ation of these categories either individually or as a lumped set of time-cast 
parameters within a more extensive activity. 


The following paragraphs provide a detailed discussion of these five 
categories of I0S activity and of how they are affected by the choice of in- 
terface devices and the layout of the [0S. 


3.2.2.1 Perception. Human performance in this function depends largely on 
the attention-getting characteristics of a display carrying new information. 
These may be related to basic readot size and intensity, dynamic behavior, 
and reinforcement by a secondary signal. 


Displays implemente? or an I0S may be described in terms of the ease 
with which the displays are noticed. The stimulus impact of the display may 
be marginal (nondescript signal in a noisy background), normal or excessive 
(display overkill). The stimulus impact of a device intended to attract in- 
structor ettention can be quartified for use in a TAM, since TAMs contain 
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some tasks known as "ATTENTION-GETTERS." In accordance with the SAINT method 
of modelling, a parameter set quantifying the length of time necessary to 
complete a task must be supplied by the user. The user first determines 
which of the above three stimulus impacts best describes the device in ques- 
tion, and second, chooses the appropriate parameter set from the parameter 
index supplied in Appendix G. Probabilities of the instructor successfully 
noticing the new display must then be assigned by the user. If the display 


is weak, a low probability is assigned, whereas a high probability of success 
is assigned to a strong display. 


Missing a change in displays due to inadequate presentation results 
in loss of time and possibie mission failures. If a change in a display is 
not noticed, the operator may proceed to another task totally omitting what 
might be an essential task in the mission. The display characteristics pre- 
sented herein apply not only to tasks which call for instructor attention, 
but also to the simple act of reading any display (¢.g., labels on push but- 
tons, positions of switches, or the data presented on CRTs). If a display is 
poor, i.e., the intensity of a CRT is low, or the label describing a switch 
* is too small to be read easily, the instructor must take additional time to 
By: read it. This decreases the time remaining for the mission and, in turn, 


increases the workload stress felt by the operator, thereby degrading in- 
structor performance. 


These display characteristics are accounted for in the TAMs. -Every 

3 task which requires the instructor to read a message is preceded by a task 

Ss which describes the display characteristics of the device being read. This 
: task is called an information presentation task (IPTASK). 


This method of describing display characteristics, IPTASK, fits wel] 
a into the TAM scheme, and into SAINT. Ergonomic literature surveys can supply 


a the data to quantify these IPTASKs, and thus, the parameter sets can be devel- ; 
i oped and inserted into the TAMs ‘to describe the effect of information presen- 
ay tation on instructor performance. 


ae 3.2.2.2 Recognition. Following perception, the newly displayed information 
1. must be evaluated and understood in terms of the status and progress of the 

ee training mission, and in terms of the next step or input required of the in- 
structor. The suitability of the I0S display device is established in rela- 
tion to the combinations of two types of information being read (qualitative 


and quantitative) and two purposes for reading (information only, and deci- 
PL: sion making). 
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The type of information being read defines basic task difficulty, 
and tthe purpose -of reading affects the work stress level because more pre- 
cision is required when data are read for decision making, than in the case 
of recording data. A single read task is used to describe all possible com- 
binations of information type and purpose. Appropriate quantifying data are 


added as parameters. In this way the read task, with appropriate parameters, 
can describe any of the fol lowing: 
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3.2.2.2 Recognition (Cont'd) r 
(a). Read/interpret moderately complex display by check-reading salient 
points. Presentation in conventional format. (Read mapping dis- 
play or graph plot for information only) 
(b) Read complex and variable display by check-reading selected values | 
and locations. Presentation in conventional format. (Scan re- 
peater flight instruments for nionitoring purposes) 


(c) .Read/interpret highly complex and variable displays by pattern re- 
cognition process. Presentation in comprehensive form. (Read tac- 
tical situation display for monitoring purposes) 


(d) Read/interpret complex and variable quantitative display by check- 
reading specific locations and determining changes. Presentation 
in comprehensive form. (Read heads-up display presentation for de- 
cision making) z 


(e) Read quantitative inf srmation composed of partial messages. Pre- 
sentation of alphanumeric data in the scrolling mode. (Read port- 
able control unit for decision making) 


The principal intent of any effort involving recognition is to fol- 
4 low the progress of the training activity and decide on the next activity. 

: From the viewpoint of instructor workload, ail changes in displayed informa- 
p.. tion are pertinent although some changes result in a decision that no action 
¥ is needed. Hence, the principal measure of success in interface design is 
how quickly the instructor can relate given information to training object- 
ives, and to the input activity required to effect the next step. The model 
assesses this success in terms of whether or not the instructor has had suf- 
ficient time to perform all tasks in such a manner that the continuity, cre- 
dibility and effectiveness of the training session are maintained. 


Recognition performance will normally be satisfactory if the follow- 
ing criteria are met: 


(f) All pertinent information is present at the correct time. i 


(g) Presentation format is easily associated with the conditions of the 
simulated training maneuver and the instructor task on hand. 
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(h) Rate of information display and dynamics are within limits of numan 
sensory and cognitive capabilities and form an acceptable workload. 


(i) Display format is conducive to quick and unerring operation of the 
input devices assigned to control the variable(s) in question. 


The last of the four is not a factor of recognition as such. It is 
mentioned here as an important component of the product of this activity 
(i.e., decision making and input action). 
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Any significant shortcomings in these areas will force the instruc- 
tor to perform additional activity, thereby increasing his workload and the 
possibility of error. 


The combination of perception (IPTASK) and reading types provides a 
means of evaluation display characteristics. SAINT provides the means to de- 
termine, in quantitative terms, the effect(s) any of these factors or their 
combinations have upon the instructor and, thus, their overall contribution 
to the effectiveness of a given IOS. 


3.2.2.3 Decision Making. Decision making is an extremely complex human 
operator process. A full treatment of decision parameters would by far ex- 
ceed the capabilities of the model and scope of this study. However, since 
flight instructors represent a select and highly trained group with above 
average qualities and skills in control and management of dynamic systems, 
the assumption has been made that the instructors will invariably make the 
correct decision if: 


(a) They are made aware of new information and pacing factors as dis- 
cussed in paragraph 3.2.2.1 


(b) All pertinent information is made available to them without any 
further effort on their part as discussed in paragraph 3.2.2.2 


(c) The display eliciting their decision has spatial and temporal cor- 
respondence with the input device to be used as discussed in para- 
graph 3.2.2.4. 


In this context, decision making includes the determination of the 
required response and the selection of the input modalities to be used. 


The analytical mode? represents decision making effort by the opera- 
tor workload and accounts for the cost of decision making in terms of the 
recognition effort required to make a simple comparison and yes or no deci- 
sion. The decision itself is almost instantaneous. The time is consumed in 
the three conditions listed as 3.2.2.3 (a), {b) and (c). Times consumed are 
products of the interface devices and the I0S layout. 
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Instructional activities, such as the assessment and documentation 
of crew performance, also include decision making tasks. Here again, the 
time costs are accounted for in the acquisition of information and the re- 
+8 sulting physical activity. 
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d The RTP requires a debriefing and critique to the trainee immediately 
5. following a maneuver. This, in turn, demands that the I0S displays enable 

+ the instructor to follow the maneuver and support on-line decisions. Acqui- 

sition of the display may involve some activities in addition to the mission 
operation and monitoring workload. 2 
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It may be appropriate to note here that the SAINT operators respond 
to conditions generating workload stress, but return to their normal perform- 
ance capability without hysteresis or residual effects as soon as the work- 
load peak passes. The characteristic may be somewhat optimistic considering 
the average human operator, but seems reasonable for the select group of 
flight instructors. 


3.2.2.4 Actions. The factors considered in the actions an instructor per- 
forms on an IOS are as follows: 


(a} Basic physical and ergonomic aspects involving visual and manual 
access to the input device, the suitability of the input method to 
the task and the resulting workload 


(b) Criticality of the input and the possible effects of an error of 
omission or commission 


(c) Operative workload generated by the display/control relationships, 
e.g., time lag inherent in machine response 


(d) Operative workload generated by task conflict, e.g., when instruc- 
tor has to participate in the simulation with radio messages. 


As in the case of displays, it was assumed the fundamental ergono- 
mic requirements and the principles of good human engineering have been sat- 
isfied in the general layout and operability of the input devices. The mo- 
delling emphasis is on the functional suitability of each device to all tasks 
assigned to it and on the operative workload in the context of a multitask 
representative task profiie. ; 


Visual access is considered necessary for every manual and reading 
task, and is included as a parameter in the TAMS, Visual access requires 
minimal time if only the eyes and head must shift, and slightly more time if 
the torso must move. However, the cumulative time consumed in a number of 
visual access tasks is not negligible and could represent a flaw in I0S de- 
signs (where a "tennis game" condition exists). Each TAM provides visual 
access tasks which precede any read tasks. By choosing the appropriate para- 
meter sets, the user can tailor this visual access task to represent either 
the eye motion only, eye and head motion, or eye, head, and body motion re- 
quired to view a display. 


Manual access nas been the subject of extensive research in indus- 
trial, vehicular and man-computer applications. Two principal conclusions 
may be drawn regarding the 10S activities: 


(e} Hand movements within the principal operating area will require a 
constant time expenditure since the arm and hand will accelerate/ 
decelerate according te the transfer distance. 


(f) The end task following the hand transfer influences the access time 
since the operator will approach a device with caution and at a 
Slower speed if the end task is critical or complex or involves un- 
usual hand attitudes. 
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Manual access tasks precede any task which calls for the activation 
of a device within a TAM. These tasks may be tailored to fit the appropriate 
IOS by the insertion of the parameter set which best describes the distance 
which must be travelled in order to access a device. 


3.2.2.4.1 Monitoring. Recognition and decision making are facilitated if 
the disolays .and controls are related to each other in a logical way. Manual 
input is invariably followed by instructor effort and time to verify that the 
correct system response is being obtained, that the machine is presenting the 
training problem as planned. A direct and dynamic machine: response increases 
confidence and reduces the overall] task time. Consider the example of an 
illuminated push button-which invokes action by changing status before the 
input and indicates acceptance of the input by a further change in status. 
The push button illumination may be pulsed when ready for actuation and fully 
lit once actuated. Further, in the case of a remote echo (cueback or verifi- 
cation display), the TAMs include a verification step. containing eye trans- 
fer and a reading task in accounting the time costs. 


The workload generated by extensive manual tasks (e.g., lengthy typ- 
ing. exercises involved in data entry) is shown by SAINT as increased stress. 
on the operators (‘in this case, the manual operator, see paragraph 3.2.3.1.2). 
The model indicates whether this task loading is enough to justify the intro- 
duction of a different input mode or automation of some functions. 


3.2.3 Analytical Network 


3.2.3.1 SAINT Provisions 


3.2.3.1.1 General. SAINT is a combined network modelling and simulation. 
technique. SAINT was designed to model, in network form, sets of tasks per- 
formed during the course of a mission. SAINT obtains mission performance 
measures for networks which represent a mission consisting of a set of tasks 
performed by a crew of operators. A simulation approach is used to obtain 
the performance measures. Human engineering considerations are included 
through parameters associated with tasks (i.e., time required to perform the 
task), precedence relations between tasks and factors affecting crew perform- 
ance (i.e@., operator speed, accuracy and stress threshold). SAINT's analysis | 
is performed by a digital computer program which performs two types of ana- 
lyses. The first analysis (benchmark iterations) obtains estimates of the 
times required by the operators to perform this mission. The second analysis 
uses the results of benchmark analysis to obtain mission performance measures 
which provide estimates of successfully completing the mission under stress 
conditions (for a detailed description of SAINT, see paragraph 2.2.3). 


3.2.3.1.2 Specific Conventions. Several conventions were adhered to 
throughout this. study to adapt SAINT to the evaluative purpose. 


The first convention is the division of one human operator (in- 
structor) into seven specialized SAINT operators. 


3.2.3.1.2 Specific Conventions (Cont'd 


This convention is necessary because, for this application of 
SAINT, we need a method of separately allocating various sensory and response 
modalities of an operator to various tasks. Modelled as one operator, the 
instructor would be able to perform only one task at any given time due to 
SAINT's structure. Thus, something as simple as speaking into a microphone 
at the same time as depressing a foot pedal would be impossible. Therefore, 
the single instructor has been divided into the following: 


(a) Visual operator (#1) 
Right manual operator (#2) 
Left manual operator (#3) 
Cognitive operator (#4) 
Audio operator (#5) 
Verbal operator (#6) 
Pedal operator (#7). 


With this configuration it becomes possible for the instructor to 
look for a switch (visual operator) while depressing a foot pedal (peda) 
operator). Several tasks may then run in parallel, and the activities of 
each modality can be tracked. 


The second convention is necessary to clarify the availability of 
the cognitive operator. Since the instructor must be cognizant of every task 
he/she is doing, and all surrounding events, this would seem to preclude 
parallel tasks because the cognitive operator cannot be in two places at once. 
This conflict appears only in the RTP (the TAMs are a straightforward series 
of tasks) where many tasks are run in parallel and the cognitive operator 
must be present for al} of them. The solution to this problem was found by 
assigning the cognitive operator to a distinct cognitive task which runs in 
parallel with the other tasks of the network. The cognitive task is an in- 
vented task used to model the instructor's mental processes employed in per- 
formance of other tasks. The effects of the instructor having to perform 
tasks in parallel are shown in the stress levels, and because of the division 
of the human instructor into seven emulated operators, the facilities of the 
instructor under greater stress can be identified. Now, the instructor may 
monitor communications (audio operator) and make written notes (manual opera- 
tor) at the same time, because the cognitive operator is performing the cegni- 
tive task in parallel with the other two tasks and is assumed to be present 
and aiding in the completion of those tasks. 


These conventions, while developed to conform to SAINT's require- 
ments, proved to be an aid in evaluating 10S designs. The division of a hu- 
man instructor into seven emulated operators allows the user to see exactly 
what is causing the stress on the instructor (i.e., the visual operator may 
be overworked while the audio operator is idle). 
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3.2.3.2 Parameters: Real World to Model. Real world concepts must be quan- 
tified in order to make them usable in the model. These concepts must be 
transformed into mathematical terms (e.g., a mean time, a minimum and maximum 
time, and a standard deviation) to describe the time taken to perform an act- 
ivity or the time taken by a characteristic or a device employed on the IOS. 
In other words, the real world terms must be transformed into basic ergonomic 
data quantifying the tasks in the TAMs before they can be used in the model. 


The categories of tasks where parameters must be quantified are des- 
cribed in paragraphs 3.2.3.2.1 to 3.2.3.2.5. 


3.2.3.2.1 Visual Access. Visual access is quantified in terms of the amount 
of time consumed in moving the eyes, head, torso or any combination of these 
in order to focus on the device to be read. Ergonomic data classify these 

times by the distances which must be covered. These parameters describe, in 
modelling terms, the real world concept ef looking from one place to another. 


3.2.3.2.2 Manual Access. Manual access tasks are classified in much the 
Same manner as visual accesses. The time needed for a parameter set is that 
time taken to move either the hand, arm, torso, or any combination of thése 
to a position from which a device may be activated. The actual activation 
time of the device is not included in this parameter set. As can be readily 
seen, I0S design and engineering data are the determining factors in the 
selection of the appropriate visual and manual access parameter sets. 


3.2.3.2.3 Device Activation. Device activation is time necessary to acti- 
vate the chosen device. These parameter sets are not IOS dependent, due to 
the fact that the hands are assumed (by completion of the manual access task) 
to be in position to activate the device. 


3.2.3.2.4 Read. Read tasks have been classified into four groups. The 
groups are defined in terms of purpose. This distinction in purpose is evi- 
dent by the length of time taken to perform the particular read task. The 
four groups of read tasks are as follows: 

Read qualitiative data for inrormation. 

Read quantitative data for information. 

Read qualitative data for decision making. 

(d) Read quantitative data for decision making. 

3.2.3.2.5 Display Characteristics. The most intricate parameters to assign 
are those describing the display characteristics. These parameters must des- 


cribe, in terms of time consumed, the effect of the display design on the 
time needed to read that display. 


3.2.3.2.6 Literature Searches. It should be noted here that literature | 

searches for the determination of the values of these parameters were beyond 
the scope of the study. Where values were easily found in available litera- 
ture, they were used. Values that were not easily located were roughly esti- 
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mated through experimentation. Members of CAE staff, totally naive to this 
study, were asked to perform functions (i.e., depress push buttons, locate 
switch positions, read lines on CRT pages, etc.) on an in-house IOS. The 
times taken to perform these tasks were recorded and averaged and used 4s 
parameter sets. These values are presented in. Appendix G. 


3.2.4 Summary. The IOS interface is a means of establishing communication 
between the instructor and the simulator software using displays and control 
equipment. The interface may be evaluated by the designer/evaluator with re- 
gard to functional aspects, operator workload capacity, task time and poten- 
tial sources of error. The evaluation method chosen is the SAINT model be- 
cause the network structure of operator performed tasks is compatible with 
the features of an IOS to be evaluated. 


The quality of an 10S can be defined in terms of the ease with which 
an operator can perform the fundamental activities: perception, recocnition, 
decision making, action, or monitoring. : 


Perception is the act of becoming aware of new information. The bene- 
fits of perception evaluation are significant. Design parameters such as 
readout size, dynamic behavior, and impact reinforcement of stimulus, and 
display qualities may be evaluated and selected with respect to the time con- 
sumption for which their characteristics are responsible. 


Recognition is the decoding of information. Recognition follows per- 
ception. Analysis of the time required for the perception activity will aid 
in the selection of optional display format. 


Decision making is a complex human process preceded by perception and 
recognition. Decision making includes the determination of the required res- 
ponse and the selection of input mode. The cognitive operator (paragraph 
3.2,3.1.2) is the only cperator capable of making a decision. 


Actions are the physical movements necessary to access a device. The 
parameters describing actions are quantifications of the time consumed by the 
manual and visual operator due to the layout of an I0S. 


SAINT iS the software chosen to support the evaluation. Using SAINT, 


_the performance of individual tasks is emulated. Both the emulated operators 


and tasks have parameter sets describing their performance in terms of time. 
SAINT’ outpi.t is given in terms of stress on the operators, times of task 
complefion and number of mission successes (i.e. missions compteted within 
the aliotted time) under variable I0S design conditions. 


These real world concepts (i.e., perception, recognition, and decision 
making) are transformed into simulation data through parameter sets. At the 
TAM level, access tasks, device activation. tasks, read tasks and display 
characteristics are assigned parameter sets found in ergonomic Jiterature, 
quantifying the times necessary to perform these tasks under different design 
conditions, 
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3.3 Objective Evaluation Methodology 


3.3.1 Introduction. To fulfill the requirement to design or evaluate an I0S 
interface, the following procedures were required: 


(a) A method by which a user could describe a real or conceptual I0S in~ 
terface in quantitative terms from real life data including an ex- 


pansion capacity to accept descriptions of new devices and condi- 
tions 


(b) Objective assessment of an existing or conceptual interface pertin- 
ent to the instructional objectives 


(c) A format for presentation of results with a quick turnaround and 


with good visibility of causative factors and potential means of 
‘mproving performance. 


A fundamental requirement was objectivity in the light of the highly 
subjective and variable assessments obtained in the past. Guidelines were 
established for the methodology. 


(d) Systematic. To model, in a step by step fashion, the activities of 
an instructor/operator during a training session and to relate, in 
a logical manner, the output of the model to the input. 


(2) Quantitative. To convert real life descriptions and subjective 
factors into objective, numerical results. 


(f) Sensitive. To produce a reasonable and comparable change in the 


evaluation results for comparable changes in the JOS design or 
operating conditions. 


(g) Repeatable. To produce similar results for similar changes. 


A further requirement was that the results of the evaluation be task- 
specific, showing the suitability of an interface device or function to an 
instructor activity. This gave rise to two discrete evaluation phases: 

(h) Static evaluation, in which the basic physical applicability and 
operability of a device or control method are assessed against a 
task, establishing step by step descriptions and time requirements 


for completion. This evaluation yields the building blocks for the 
next phase of the evaluation. 


(i) Dynamic evaluation, in which an I0S design, composed of the build- 
ing blocks yielded from the static evaluation, is tested against a 


Hl representative task profile with many concurrent and possibly con- 
7 flicting events. 
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An essentially related effort was to systematically describe the tasks | 
and activities which could be assigned to the instructor. The principal goal 
was to guide the user from qualitative descriptions of instructional tasks to 
definitive specification and detailed analyses of all the activities involved 
in their execution, in quantitative terms suitable for insertion in the 
model, This was accomplished in the formulation of four developmental steps: 


(j) The Instructional Task List (ITL) describes instructional tasks in 
generic terms. | 


({k) The Representative Task Profile (RTP) comprises a set of typical 
training activities arranged to present tasks and workload levels 
similar to those of a training session. 


(1) The Instructor Task Description (ITD) supports the analysis of the 
typical tasks in terms of information exchanges at the IOS and 
thereby reflects the results of the man-machine task allocation 
scheme applicable to the 10S in question. 


(m) The IOS Activities List carries details of the elemental tasks and 
relates these to the use of a given I0S device or function. It 
also references TAMS which relate the activities required to per- 
form a task to the device used. 


3.3.1.1 Instructor Task Descriptions. An analysis of the items of the ITD 
(a sample is provided in Figure 3-1) revealed that a set of 19 descriptions 
will cover all the tasks in the RTP. These tasks are: 


(a) Monitor crew activities 
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(b) Display management 

(c) Reset malfunction 3 
(d) Arrays display q 
(e) Simulate surface facilities and communications ] 
(f) Brief crew and demonstrate maneuver 3 
(g) Assess crew cooperation in combat maneuvers y 
(h) Assess accuracy of verbal communications | : 
(i) Automatic insertion of discrete malfunction | ‘ 
(j) Critique crew performance | 

(k) Activate discrete malfunction 


(1) Simulate ground support activities and communications 
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Figure 3-1 Typical Page from the ITD List 
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Simulate target conditions and combat activities 
Assess accuracy of navigation and ground track control 
Assess crew performance in combat maneuvers 

Assess accuracy of discrete activities 

Document assessment of crew performance 

Assess sequence of discrete activities 


Assess crew performance in continuous control activities, vehicle 
handling and flight path control. 


3.3.1.2 Typical Activity Moduies, The TAM library provides the necessary 
modelling details to represent almost any given IOS activity listed in the 
ITD. Furthermore, the tasks within the TAMs may be made active or dormant, 
an approach which permits flexibility and generalization in definition, 


This modular, or building block, approach enabled us to develop an 
analytical model which the user can easily manipulate to conform to an exist- 
ing or conceptual IOS without expertise in software or computer methods. 


3.3.2 Instructional Task Delineation 


3.3.2.1 Instructional Task List (JTL). The first necessary step was to list 
all known activities required to: 


(a) Set up the simulator and define the simulated conditions for the 
training session. 


{(b) Control and actively participate in the presentation of the train- 
ing problem. 


(c) Observe and assess trainee progress and proficiency levels. 
(d) Document the training session. 


In most cases all of the above involve inputs and activities at the 
IOS, and all necessary dispiays and controls must be available. However, 
there is a significant difference between the preparatory phases and the dy- 
namic on-line periods of the session, particularly in terms of the pacing of 
events and the resulting operator stress. 


The Instructional Task List (ITL) (provided as Appendix A) describes 
the activities of preparing and conducting a training fiight, such as setting 
up ambient weather conditions, some of which are peculiar to simulator train- 
ing. The description of activities was developed on the basis of the lesson 
plans of over 100 simulators of various types and represent collective exper- 
jence with a variety of instructor facilities. Reference to the IOS inter- 
face method or the device that may be used to perform a task is intentionally 
avoided to achieve flexibility and to encourage assessment of hardware, 
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The ITL provides the designer/evaluator with a checkist by which the 
overall training objectives may be resolved into their component steps and 
activities in a systematic manner, Figure 3-2 shows a sample ITL page along 
with the explanatory legend. 


3.3.2.2 Representative Task Profile (RTP). Since, in this evaluation metho- 
dology, the criterion for success of an IOS design is whether or not it en- 
ables the instructor to perform all essential tasks within available time, 
and since the instructor workload is the key to effective measuring of an in- 
terface device, it was necessary to construct a typical task profile against 
which the candidate I0S interface methods could be evaluated or compared in a 
quantitative fashion, 
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A typical training mission has been assembled with a reasonable se- 
quence of events selected from all sections of the ITL. The operative dis- 
play/control tasks have been arranged to resemble the phases of the training 
flight. Continuous didactic tasks, such as monitoring of the trainee and ee 
performance assessment, have been added ‘to the appropriate mission phases. A 
pattern of malfunctions was activated both in the manual and in the prepro- my: 
grammed mode. The simulated flight was interrupted at certain points to de- i 3 
liver briefings, critique and demonstrations where this could be done without 
destroying the continuity and credibility of the presentation. 


The RTP (Appendix B)-represents-both high and low instructor work- 
load periods, but in its basic form it is still independent of the I0S inter- 
face methodology. It describes the tasks and events in real world language. 
However, descriptions are so structured that they may be directly related to 
typical activities and tasks defined later which, in turn, can be selected to 
reflect the I0S functions and features to be used. 


The representative task profile has served multiple purposes. 
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(a) As an initial task sequence to model activities on a hypothetical 
I0S and to validate that model. 
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(b) As a standard sequence or benchmark task against which candidate 
I0S methodology and equipment could be prepared. 


nd 


(c) To support the development of a method whereby real world and eng- 
ineering descriptions could be transformed into modelling terms and 
parameters in a systematic fashion. 
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Performing the task sequence represented by the RTP presents the 
instructor with workload peaks, task conflicts, interim periods and an over- 
all task duration similar to those of a typical training session in terms of 
frequency and intensity of workload fluctuations. 
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Figure 3-2 Sample ITL Page 
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In its specific form, where references to IOS functions and devices 
have been added, the RTP closely approximates a one-hour training session 
planned for a military simulator of a jet aircraft. The I0S of this simula- 
tor is heavily supported by software and CRT displays, with direct action 
controls provided for functions such as mapping, plotting, recording func- 
tions, preprogrammed problem presentation, and communications channels where 
timing is essential or where instructor convenience is improved. In its pre- 
sent form, the RTP is heavily biased toward step-by-step manual operation, 
although muny automatic functions are available. Automatic functions could 
be used to alleviate workload through a redistribution of tasks or a retrofit 
of the I6S design. 


Figure 3-3 shows a sample page of the RTP. This pége represents the 
instructor tasks during leg 7 (roll-in and shutdown of the aircraft) of the 
total mission. The ITL column on the page shows the location of these tasks 
in the ITL. This example will be elaborated upon in each of the fuliowing 
two sections and provide a clear picture of how the RTP was used. 


3.3.2.3 Instructor Task Description. Selected items from the ITL have been 
assembled to form an RTP. The items of the RTP, such as, ASSESS SEQUENCE OF 
DISCRETE ACTIVITIES, were further broken down in the ITD into their component 
tasks (e.g., obtain information by observing crew activity, verify against 
standard procedure, note results of assessment) in order to reach a level of 
detail where repetitious statements could be used as elements in the descrip- 
tion of a task or event. The right-hand side of an ITD page (Figures 3-4 to 
3-8) provide a menu of the various ways an instructor can perform tasks on an 
0S. Not every item in the menu applies to every IOS design. The user need 
only check the menu items which pertain to the I0S being evaluated. 


The RTP was composed of repetitions of 19 tasks. These tasks have 
been expanded into the ITD and are now considered modules. This modulariz- 
ation greatly aided the development of the SAINT-RTP network (paragraph 
3.3.6.2). Each task in the ITD module menu has been assigned a permanent 
task number which corresponds to that task in the main SAINT network. The 
task applicable to the IOS under study is checked A (active, as opposed to D0, 
dormant) by the evaluator, thus making the module IOS specific. This form 
serves as a record of the activities included in the module. 


Figures 3-4 to 3-8 show the expansion of Figure 3-3 into an ITD. 
The checkmarks indicate which menu items have been activated to emulate how 
the ITO heading would be performed on a jet trainer I0S. 


3.3.2.4 IOS Activities List. Finally, the descriptive chain has been com- 
pleted with a listing which links the instructive tasks of the RTP and the 
man-machine task allocation of the ITD to the available/desirable IOS devices 
and functions. The I0S Activities List {Appendix E) contains a description 
of the interface devices, a brief description of how the device operates, and 
references the TAM (which contains the model of the operator use of the de- 
vice). Figure 3-9 is a sample of the list. This is the last link of the 
descriptive chain. The user need only match the tasks from the ITD menu with 
the device on the I0S (described in the Activities List) and select the TAM 
indicated. Then the user inserts the proper parameters into the TAM and runs 
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SAINT. An analysis of SAINT's output will give the evaluator data as to the 
suitability of the device to perform the task. 


For example, if the user is evaluating the I0S using leg 7 of the 
RTP (Figure 3-2) and is trying to make ITD Ref. 7.2.3 (Figure 3-8) I0S 
specific, the user checks tasks 90, 91, and 93 active. In order to perform 
task 91 (nard copy CAM display page), the evaluator will have to check the 
Jet station. This check shows that hard copies are obtained via push but- 
ton. The user then goes to the Activities List (Figure 3-9) and finds the 
description of push buttons best describing those on the jet trainer 10S and 
finds the TAM (20) that contains the model of the actions necessary to per- 
form task 91 on a push button. The user then inserts the appropriate parame- 
ters into TAM 20, runs SAINT with that TAM 20 data and obtains the static 
phase of the evaluation. The insertion of the data gathered from this evalu- 
ation into task 91 in the RTP model will yield the data to be analyzed for 
the dynamic phase of the evaluation. 


3.3.3 Evaluation Guidelines and Criteria. Advantages of tie live model 
approach to the evaluation of the design of an 10S may be summed up as fol- 
lows: 


) The mode) establishes repeatable conditions of testing 

(b) The model accepts quantitative, objective performance measures 

(c}) The method uses standardized criteria of performance 

(d) The method produces repeatable results and detailed documentation, 


Tne parameters of evaluation can be divided into three classes as fol- 
lows: 


(2) Independent Variables. [Independent variables are training object- 
ives and instructor task definitions which may be set for any re- 
presentative profile. Interface methodology and devices which may 
be selected independent of other factors are also independent vari- 
ables. The independent yariables were held constant during the de- 
velopment of the modelling network, and the interfacing methodology 
parameters were varied during the validation tests both in an arbi- 
trary manner and in a controlled mode by redefining the interface 
devices. 


(fF) Depertent Variables. The principal dependent variables are time 
avai.able to perform all tasks and the human operator performance 
characteristics. 


{g) Control Variables. The control variables are the parameter sets of 


ee 


the modeiling network and are defined by ergonomic data and/or real 
world experience. These variables carry the principal means of 
controlling the model. 


BR As general guidelines, it was established that the basic tasks should 
, be defined individually, and intrusive conditions (e.g., display formats, in- 
formation types presented) should be modelled as tasks performed by the mach- 
ine (no operator associated with the task). Thus, these characteristics are 
treated as any other tasks with timesaving, or time-consuming attributes. 

All tasks are modified by their parameter sets. 


BR Evaluation criteria have been derived from performance parameters and 

‘ operating conditions. The principal parameter of evaluation is time, there- 

fore, the principal criterion of evaluation is the time to complete all tasks. 
Another criterion is the availability of information. This relates 

the I0S devices and functions to the parameters of performance and avoids 

such ambiguities as the lack of a display device which could be falsely shown 

as saving in time since it does not need to be read. 


Although these two criteria represent the cardinal metrics of evalu- 
ation, the model provides these further details: 


(1) Attention-getter 
(2) Display format 
(3) Accessibility and frequency of visual relocation. 

(i) Criteria of Control Harmony 
(1) Layout of devices, frequency of manual access 
(2) Input task complexity 
(3) Dynamic system response check, e.g., local cueback. 

The model, especially the TAMs, provides input ports for other para- 


meters through SAINT's capability to easily manipulate parameters for the 
evaluator. 


(h) Criteria of Suitability for Information Exchange 
3.3.4 Evaluation Methods | 
qe 3.3.4.1 Static Aspects. Static evaluation covers the basic suitability of | 
an IOS device or function for a given task defined in the ITO. In other 
words, the static phase of an evaluation analyzes the availability of basic 
IOS interface devices to effect the necessary man-machine information ex- 
change. The process consists of the following steps: 


i (a) Correct interpretation of the physical layout from engineering data 
a: and the factors possibly affecting the task time. 


q (b) Systematic assessment of visual and manual access conditions. 
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(c) Assessment of conditions of information exchange, availability of 
displays and operability of controls. 


It is assumed that the fundamental human factors and ergonomic re- 
quirements have been satisfied and therefore accounted for in the data used 
to establish the task times. 


The static phase of the evaluation is carried out at the TAM level. 

It is at this level that the ability of an interface device, and its place- 
ment on the I0S, to have a particular task performed in it is tested. An 
example will more clearly illustrate the concept of static evaluation. It 
should be recalled here that TAMs model isolated functions with no regard to 
a lesson plan or surrounding activities (paragraph 2.2.3.6). If an evaluator 
wishes to determine the feasibility of calling up a map on the left CRT, scal- 
ing the map, and centering the aircraft on the map using the auxiliary key- 
board (AKB) situated on the right side of the IOS, the evaluator would choose 
the KEYBOARD TAM. The evaluator would then read the engineering data to de- 
e termine distances to be traversed both visually and manually. Armed with 
these data, the evaluator would choose the appropriate parameter sets (as dic- 
= tated by the engineering data}: from the listing of Parameter Sets (Appendix 

: G) and assign then to each task in the keyboard TAM, The final parameter to 
be decided on is the time to allot for the performance of this task on the 
particular interface device. When that is done, these jata ar* processed by 
SAINT. From SAINT's output, the evalu. tor can determine whether the AKB is a 
Suitable interface method for calling up, scaling and centering maps by ana- 
lyzing the number of successful missions, average time of completion, stress- 


£ es on the operators, etc. 

ES The data obtained in static evaluation are thus valuable in them- 
a selves and, aS will be seen, are an integral part of the dynamic phase of 
* evaluation. 


Ze 3.3.4.2 Dynamic Aspects. The individual task device combinations must be 
4 tested in the context of the IOS interface as a whole under realistic condi- 
tions involving multiple simultaneous tasks and activities. This testing 
or constitutes the dynamic phase of the evaluation. The main analytical model 
ae (i.e., model of the RTP) is the tool used for this phase. 


a: Data obtained through the static phase of evaluation (i.e., average 
ay: time of mission completion and the probability of success) are inserted into 
the model of the RTP. In conjunction with the example presented in paragraph 
3.3.4.1, the average time values of the keyboard TAM will be inserted into 
the main model each time maps must be called, scaled, and centered. This TAM 
is no longer isolated, but it now affects and is affected by other TAMs in 
this representative lesson plan. From this dynamic model, the effects of ail 
aspects of the 10S, in conjunction with TAMs and the application of the I0S 
to the lesson plan, are set forth in terms of mission success, times of 
completion and stress levels on the operators. It may be found, for example, 
H that calling maps via AKB causes higher stress on the operators during the 

. RTP mission than using a voice controller or push button. It is this kind of 
3 information which aids a designer in developing a new I0S. 
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3.3.5 Model Construction - Typical Activity Modules. The tasks identified 
by the task analysis, in this case the RTP, as well as expanded into modell- 
able events whereby each action of the instructor, and every I0S equipment 
task performed, is accounted for. Repetitive groups of such events can be 
described by TAMs. A TAM describes a typical instructor/10S activity in suf- 
ficient detail so that most task requirements may be matched to most 10S con- 
figuration details by selecting the appropriate TAM, thereby avoiding the 
work of individual analysis. The advantage of expanding these tasks into the 
form of TAMs is that relatively few TAMs comprise a library of "building 
blocks". They may be selected to describe a wide variety of I0S activities 
without repeating the analytical process. 


TAMs describe each action an instructor performs to accomplish a task 
with a designated interface method. The TAM title identifies the interface 
device being used. For example, if a decision on whether a toggle switch or 
a dedicated push button would be more advantageous in a particular 10S, the 
toggle switch TAM and the push button TAM can be run. The more advantageous 
device will be thus shown. However, the impact of each device on the imp le- 
mentation of a lesson plan can be shown by replacing the toggle switch TAM 
with the push button TAM throughout the RTP and analyzing the output. The 
comparison of output from RTP simulations using each device will show whether 
the different wevices do indeed affect the instructor workload. 


A typical TAM and its SAINT-defined flowgraph are shown in Figures 


3-10 and 3-11. The TAMs are supplied for the user with a grid in which tasks 
can be checked as either active or dormant. 


Stepping through the TAM, the user can decide if an attention-getter 
is necessary for the task at hand. Task 10, with its parameter sets, accounts 
for the time it takes for the visual operator to focus on a disptay. The 
IPTASK allows the user to account for the display characteristics by choosing 
the appropriate parameter set. The type of information displayed and its 
time consuming effects are accounted for in tasks 50 to 53. The amount of 
time taken to read the display is accounted for in tasks 60 to 63. Having. 
obtained this information, the SAINT operator must decide whether or not to 
change the display. If not, task 80 will branch to the end of the network. 
If so, the visual operator must now focus from the display to the keyboard 
(task 11). The manual operator must move to the keyboard (task 20) and key 
in the proper code. As can be seen from the Flowgraph, task 30 branches back 
to task 10. The reason for this repetition of tasks is to allow the instruc- 
tor to check that the display called up is the correct one before pressing 
the INSERT button. If the evaluator is sure that the instructor will not per- 
form this check, he/she can specify to SAINT (see User's Guide, Appendix I) 
that the parameter sets of these repeated tasks be set to zero (i.e., they 
will not use any time allotted to the mission). The remainder of the TAMs, 
and their flowgraphs, can be found in Appendix F. 


3.3.5.1 TAM Philosophy. The events that can be modelled in the TAMs are div- 
ided into two categories: constant and device-specific. Some activities re- 
quire the same amount of time to perform regardless of the device; hence, it 
is the number of times they are performed that spells out the difference in 
the 10S interfacing methodology. The device-specific tasks reflect the mech- 
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Figure 3-11 Sample TAM Flowgraph : J 


anization of the I0S and the entire interfacing methodology. TAMs have been 
developed for such devices as the light pen, light gun, push buttons, rotary 
switches, potentiometers, keyboards, and toggle switches. 


TAMs have two basic functions, They are building blocks for the RTP 
and also independent evaluative tools. In its present configuration, SAINT 
has the capability of processing 200 tasks (declaring SAINT's arrays to 
larger dimensions enhances the number of tasks it can process). However, the 
model of the RTP, using each TAM in its full form, far exceeds 1,000 tasks. 
Although SAINT can be made to process these tasks, the network would be a 
mass of confusion. SAINT itself provides a method of circumventing this pro- 
blem, By utilizing SAINT's mission summary output, it is possible to reduce 
a TAM to the status of a single task (for use in tne RTP), with parameter 
values which are supplied by the mission summary... The TAM can be inserted 
into the RTP as a single task which contains time parameters indicative of 
the entire TAM. In this manner, a complete leg cf a representative mission 
can be established as a single network comprised of condensed TAMs and exer- 
cised to evaluate the particular I0S in a representative flight regime. 


TAMs may also be used to evaluate devices independently. Using 
engineering design data to choose the appropriate parameter sets, the desir- 
ability of two candidate interface devices for an IOS can be evaluated in 
terms of time taken to perform a task and the workload stress placed on the 
operators. This evaluation can be carried a step further: if a leg of the 
RTP is run and the analysis of the output reveals that stress is high in a 
certain group of tasks, those condensed TAMs can be identified, expanded, and 
analyzed and changes implemented. The revised expanded TAM can then be sum- 


marized and inserted back into the RTP. Results of the changed tasks in the 
TAMs will be seen, 


3.3.5.2 Structure. TAMs are designed in a generalized form to account for 
every possible action that may be needed. This structure is made possible by 
allowing a task to be in one of two modes. dormant or active. Because every 
event is accounted for in the TAM, not all of these events will be needed in 
each evaluation, Those extraneous tasks can be assigned a task time require- 
ment of zero and considered dormant. Setting the time parameters equal to 
zero causes the dormant task to use zero seconds of the time allotted to the 
mission while remaining in the network. If a task is not needed for the pre- 
sent evaluation, the TAM need not be revised. Rather, the values of the para- 
meter set for the task need only be set to zero, thus maintaining the integri- 


ty of the TAM. An active task has a parameter set equal to something other 
than zero. 


TAMs are structured as a series of tasks performed by operators 
which flow through the networks. The branching from tasks is mostly determ- 
inistic, with few probabilistic branches. The probabilistic branching occurs 
From tasks which involve decision making. Decision making tasks consist of 
verifications that data presented are correct or the decision to change the 
present display. One of the branches is the correct branch, i.e., the branch 
taken if the task is performed correctly. The other branch is the incorrect 


branch, which leads to either the last task of the mission or back to the 
procedure which was incorrectly performed. 
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Attention-getters are considered to be equipment tasks. That is, 
they are treated as any other tasks. They consume time, they affect the mis- 
sion (especially if they are not noticed by the instructor), but they are 
performed by the 10S. Therefore, no emulated operator is assigned to them. 
These attention getting tasks are modelled as probabilistic tasks. The 
values assigned to the branches are the probabilities of the instructor no- 
ticing the signal. If the instructor misses the cue, the task branches to 
the end of the TAM, skipping all the interim tasks necessitated by the atten- 
tion-getter. If the attention-getter is not necessary, the task is assigned 
a parameter set where values are equal to zero (i.e,, made dormant) and a 
probability of 0.99 is assigned to the correct branch. The rationale for the 
assignment of 0.99 instead of 1.0 is due to a limitation of SAINT. If a pro- 
babilistic task branch is assigned a value of 1.0, SAINT interprets it as a 
deterministic task, but all the task decriptors state that the task is pro- 
babilistic and a SAINT error occurs. 


3.3.5.3 Tasks. Tasks contained in the TAMs can be divided into the follow- 
ing categories: 


) Visual access 

) Manual access 

) Operation 

) Information type 
e) Read 

) Decision 

) Display qualifier 

) Attention-getters 

) Display delays. 
Access tasks are those which take an operator from his/her present 
position to the position required to perform the next task. Visual access 
can be further divided into three subdivisions: 

(j) Eye movement only 


(k) Eye and head movement 


(1) Torso movement. 


Each of these subdivisions has a parameter set associated with it. 
Eye movement occurs only when the visual operator travels a distance less 
; than or equal to 12 inches (30.5 cm). Head and eye movements are usually 4 
. needed to access a device between 12 and 24 inches (30.5 and 61 cm) away. | 4 
- Distances greater than 24 inches require torso movement. Visual and cog~ F 
nitive operators are assigned to these tasks. 
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3.3.5.2 Tasks (Cont'd) 


Manual access is similar to visual access. The subdivisions are the 
same, but the parameter sets differ. Manual access does not include the acti- 
vation of the device, only the movements of the operator to a position from 
which the device can immediately be activated. Manual, visual and cognitive 
operators are assigned to these tasks. 


The visual and manual access tasks are key factors in the evaluation 
of an IOS. If an 10S is designed in a manner which requires many repetitive 
motions to access devices, the operator will show a higher stress level and a 
mission will require more time than will be the case on a better IOS layout. 
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Activational tasks are those which cover the actual manipulation of 
a device, e.g., depressing a push button or rotating a potentiometer. There | 
is a parameter set describing the time requirements for the manipulation of 


es Sitter hen, 


each device. Thus, in an evaluation or in a design, different devices may be 
inserted into the TAM and the effect of each device on mission time and oper- 
ator stress can be calculated by SAINT. Manual, visual, and cognitive opera- 
tors perform these tasks. 


Four types of information capable of being displayed nave been iden- 
tified. Information Type I consists of groups of two to four words of 1.5 
syllables/word. Information Type I is usually seen in the form of labels on 
or around activational devices. 


Information Type If is opaque and displayed on a CRT or in printed 
form. It consists of a limited, repeatable vocabulary containing words aver- 
aging 2.2 syllables/word. Any page, such as malfunction page, lesson plan, 
or initial conditions, falls into this category. Visual and cognitive opera- 
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tors are assigned to these tasks. 4 
Instruments, either repeater or synthetic where only gradations and ! ’ 
numbers on the instruments are used, comprise Information Type III. ot 
Information Type IV includes pictorial displays, i.e., maps, landing " 


approaches, etc. The parameter sets describing these information types ac- 
count ‘for only the actual act of reading the information. The reason for 
reading, or the thoroughness with which they are read, are accounted for in 

read tasks. Cognitive and visual operators perform these tasks. 


Reading tasks used in the TAMs are those described in paragraph 
3.2.2.2. The visual and cognitive operators are assigned to these tasks. 
Decision tasks used in the TAMs are explained in paragraph 3.2.2.3. The cog- 
nitive operator performs the decision task. 


Dispiay qualities are accounted for in an information presentation 
task (IPTASK). In keeping with the scheme of the TAM, the IPTASK immediately 
precedes the information type task, which immediately precedes the read type 
task. Thus, any device to be read is quantified by its display characteris- 
tics. These include: 
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3.3.5.2 Tasks (Cont'd) 


m) Luminance 
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n) Contrast ong 
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Flicker 


) Color 5 
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(q) Resolution 

(r) Size 

(s) Character generation method 
(t) Surrounding area. ? 


Time values for each of the parameters are to be determined by the <i 
user (either through data searches or by experiment) in developing a parame- . 
ter set for an IPTASK. A display where applicable characteristics take zero a 
time away from the reading of the device (i.e., a good display) will be as- “2 
Signed a parameter set where mean, minimum, maximum, and standard deviation 
values are zero. If one of the characteristics is poor (e.g., contrast), ad- mS 
ditional time will be required to read the display. Therefore, the mean ¥ 
value in the parameter set shall be the average amount of time that charac- 
teristic consumes. If atl the characteristics considered in the IPTASK sum 
up to an average display (i.e., the mean of the parameter set equals zero) 
and then one of those characteristics is improved until it greatly helps the 
quality of the display, the IPTASK will be assigned a negative mean time in 
the parameter set. A negative mean time for a task causes the expected end 
time to be lower than that for its predecessor task. For example, if the end 
time of the predecessor of the IPTASK is 130.00 seconds into the mission, and 
the contrast characteristic of a display has been improved until it is better 
than average, the IPTASK parameter set will be assigned a mean of -1.0. 
Thus, the expected end time of the IPTASK will be 129.00 seconds, thereby a 
a adding an extra 1.0 second to the time allowed for the mission. q 
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Attention-getting is another equipment task. If an attention-getter 
is needed for the running of the TAM, a parameter set, with a value calculated 
in a similar way to that of the IPTASK, and reflecting the same characteris- 
tics, is assigned to the task. The probabilistic branching from the atten- 
tion-getter reflects the probability that the attention-getter is noticed. 4g 


Another task seen on some TAMs is the display delay. This is the 


iy: time delay between activation of the device and feedback. Obviously, a high a 
a. time cost in a display delay causes time to be taken from the remainder of 
the ri on. 
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3.3.6 Modular Network Construction. Through the development of the ITD and 
the IOS activities lists, the frequency with which some instructor tasks oc- 
curred during the RTP was determined. The 10S Activity List shows the reso- 
lution of these instructor tasks into SAINT defined tasks, or subtasks, In 
the real world, several tasks were performed simultaneously by various opera- 
tors. Thus, parallel representation of some tasks was necessary. 


Figure 3-12 shows the module entitled ASSESS SEQUENCE OF DISCRETE ACTI- 
VITIES. This is the SAINT flowgraph depicting the ITD sequence of actions 
shown in Figure 3-13. The numbers within the nodes of the flowgraph are not 
relevant to this discussion, but are explained in the User's Guide, Appendix 
I. The right-most number in the node corresponds to the task with the same 
number on the ITD (Figure 3-13). Tasks 42 and 48 have no descriptors in the 
ITD. This is because they are dummy tasks (i.e., represent nothing and have 
zero value parameter sets) and are present only for branching purposes. They 
ensure that all operators are present to perform the tasks of the module, and 
ES that all leave the module to go on to tine next module. The triangles in the 
a network represent an operator and the number inside the triangle identifies 
the operator. Tasks shown in parallel (Figure 3-12) are considered to be 
running simultaneously. Thus, a human instructor divided into emulated oper- 
ators (visual, manual, audio, cognitive: see paragraph 3.2.3.1.1) can moni- 
3 tor intercom exchanges (task 44, operator 5), make written notes (task 45, 
ee: operators 2 and 3), read crew action monitor (CAM) and standard procedure 

checklists (tasks 43 and 46, operator 1) simultaneously. Parallel tasks have 
= a strong effect on the workload stress of the operators. As explained in 
Ei paragraph 3.2.3.1.2, the mental processes of the instructor, represented by 
F the cognitive operator (operator 4) performing a cognitive task (task 47), 
a continue throughout the duration of the module. 
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EB. The user annotates each task sheet to indicate whether the subtask is 
Be active or dormant. These groups of tasks are then arranged through proper 

‘ branching to form the model of the RTP. This modular method has several 
clear-cut advantages from the user's viewpoint. First, the integrity of each 
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fi. module, its flowgraph and its task descriptors, remain intact. The entire “4 
; set of subtasks comprising the ITD item will always be presented together, us 
at although only a few of those subtasks may be required. Second, this modular- . 


ization of the subtasks allows the user to develop a much smaller network. 

Instead of repeating each ITD item as it is required (up to four times during ; 
some legs of the RTP), the user merely branches back to the required module | 
as often as necessary. 


swt c soateteetat! Aborted Evers 


Whee 
its vehe 


Ceaetaveren 


This repeatability has another advantage. Due to the fact that the 
modules are repeated instead of rewritten (which would entail assigning a new 
{ 


identification number to the same task each time it is rewritten), it becomes 
Sf possible for the user to determine the sum of the times taken to perform a 
tai. particular task by locating onty one task number. This is opposed to search- 
ing the printout for the same task represented by several different numbers. 
For exainple, the ITD, ASSESS CREW PERFORMANCE IN CONTINUOUS CONTROL ACTIVI- 
TIES, VEHICLE HANDLING AND FLIGHT PATH CONTROL occurs five times. One of the 
subtasks, or I0S activities of that ITD, is task 59, which is MAKE WRITTEN 
NOTES. The amount of time spent making written notes during the five times 
the crew performance was assessed can be tallied simply by summing the times 
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Figure 3-13 Sequence of Actions (ITD) 


NOTidIYISIG ASVL YOLINULSNI 


Me Sk eT 


TONE TTP Segoe cake x’ ert se * 


2 hax de 


Petite ake 


Aa ine 6 Sie tt ast eae earned held se, 
* . DES Oe RGA Magre ett: 
. Co 


ere eR ee Lee ee SM NE fg otra me 


task 59 is used. Were it not for the modularity of the ITD, the task MAKE 
WRITTEN NOTES would have appeared five times with five distinct identifica- 
tion numbers, causing the user considerably more difficulty in locating the 
times for these tasks. 


The complete set of module flowgraphs, along with the SAINT data for 
each module, is presented in Appendix D. 


3.3.6.1 General. There are 19 basic ITD items which comprise the instructor 

tasks of the RTP and are repeated throughout the RTP. The subtasks of these 
items have been flowcharted to SAINT specifications and remain uncorrupted in 
any subsequent modelling. Evary subtask comprising each ITD item has been 
assigned an identification number which should not be changed by the user. 

The branching from task to task in the flowchart of the modules should also 

be left intact. An explanation of the symbology will follow. io: 
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Also associated with the flowchart of each module is the data des- 
cribing each task comprising the block. It is here that the user controls 
the model by inserting the desired parameter sets and time perturbations 
which describe the tasks. 


For example, ITD item SIMULATE SURFACE FACILITIES AND COMMUNICATIONS 
can se used to illustrate the principles followed in fiowcharting (see Figure 
3-14). Tasks 95 and 102 are dummy tasks, inserted solely for branching pur- 
poses, They do not represent any IOS activities. Task 95 collects all seven 
operators and readies them to release their designated tasks. Task 102 col- 
lects all seven operators after they have completed their tasks and sends 
them to their next designated task via CONDITIONAL, TAKE FIRST BRANCHING. 

Gue to the parameter sets assigned to these tasks, they use zero time from 
the allotted mission time. The task numbers correspond to task titles set 
forth in the ITD (Figure 3-15). All operators gather at task 95 and then 
branch through the module. Upon completion of task 88, CHECK LESSON PLAN, 
the audio operator proceeds to task 90, MONITOR INTERCOM, the verbal and foot 
operators branch to task 91, USE IRTERCOM, and the cognitive and visual oper- 
ators proceed to task 89, SCAN MAP DISPLAY. An assumption had to made ‘here 
which will be continued through tre rest of the modelling. 


Due to the nature of SAINT, one operator cannot perform more than ' 
one task during the same time interval. However, from a practical viewpoint, 
the cognitive operator must be an integral part of each task. The modular 
approach offers a solution to this problem. If the mental operator is kept 
occupied through the module, it can be assumed that it is present and parti- 
cipating in each task. 


The final task (task 102) in this network, as in all others, has 
CONDITIONAL, TAKE FIRST BRANCHING. This branching fits well in the scheme of 
modularity. By setting the branching conditions properly (this is already 
set for the user), branching will occur to the various moduies as often as 
indicated without having to repeat the module as another entity. 
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Figure 3-14 Module Flowc 
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3.3.6.2 Module Classification. The mudules which form the RTP are divided 
= into three classes. 


a 3.3.6.2.1 Simulation Control Modules. These consist of all the actions an 


a instructor must perform in order to run a simulation mission. Control mo- 
A: dules are: 


(a) Manage displays (i.e., calling up CRT pages, maps, etc). 

(b) Brief crew and demonstrate maneuver. 
‘ (c) Simulate surface facilities and communications. 
: (d) Simulate ground support and communications. 
Eo (e) Simulate target area conditions and combat activities. 
i (f) Activate discrete malfunctions. 

; (g) Insert discrete malfunctions, 
RY 3.3.6.2.2 Assessment Modules. While having no effect on simulation, these 
fo are an integral part of the instructor work during a training mission. The 
E instructor must assess trainee performance in various facets of flight. The 


2 IOS design and hardware have a great effect on the workload which the perform- 
ance of these tasks imposes on the instructor. The assessment modules are: 


(a) Assess accuracy of verbal communication 


3 (b) Assess crew performance in continuous contro) activities, vehicle 
+ handling and flight path control 


(c) Assess accuracy of navigation and ground track contro) 
i) (d) Assess sequence of discrete activities 
i e) Assess accuracy of discrete activities 
) Assess crew cooperation 
) Assess crew cooperation in combat maneuvers 
) Assess crew performance in combat maneuvers 
(i) Critique crew perforinance. 
*,3.6.2.3 Documentation Modules. These afford the means of making formal 


written notes, hard copying the crew actions monitor display, hard copying an 
approach plot, or transferring data to disc files. 


| 
ig 
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Each assessment module contains a task record of MAKE WRITTEN 
NOTES. This does not take the place of MAKE FORMAL WRITTEN NOTES in the do- 
cumentation module. The notes taken during an assessment are merely "scrib- 
bles or jottings" which are often intelligible to only the instructor and 
used in the writing of formal notes. 


3.3.6.2.4 RIP Flowgraph. Each division of modules was assigned a position 
in the RTP, The simulation control] modules occupy the top line of the RTP 
flowgraph while the assessment modules and documentation modules occupy the 
second and third lines. This aids the user in quickly identifying the type 
of task being performed. 


Figure 3-16 (from Appendix F) represents the flowgraph of RTP leg 
4, implementing the module classifications, Each block represents a module 
of the ITD (to present the ITD module in its full form would increase the 
size of this drawing tenfold, making it cumbersome and confusing). Each oval 
represents a display management task. The top line of the flowgraph contains 
all simulation control tasks. The second line contains the assessment and 
display management tasks needed only for assessment purposes. Line three con- 
tains the documentation tasks. Figure 3-17, the verbal representation of leg 
4 of the RTP, may be used as an aid in interpreting the flowgraph. 


3.3.6.3 Model Parameters. Each task in the network has a set of task des- 
criptors or parameters associated with it. A brief synopsis of each descrip- 
tor will be given here, and a full explanation is provided with the User's 
Guide, Appendix I, The task number is the identification number of a task in 
the network. No two tasks can have the same identification numbers. Task 
type is a code number indicating whether the task is performed by a single 
operator, joint operators, or by the equipment itself. Branching character- 
istics are code letters indicating whether the operators branch deterministi- 
cally, probabilistically, or conditionally to another task. Release require- 
ments indicate the number of tasks which must be completed before the task at 
hand is released (i.e., begun). There are two release requirements: one is 
the prerequisite for the first release of the task, and the other sets the 
requirements for any subsequent release of the same task. The parameter set 
number i¢ the identification number (usually the same as the task number) of 
the paraneter set associated with the task. Task class is an identification 
number between one and ten which identifies what group the task falls into. 
Figure 3-18 shows the descriptors as they will appear in a SAINT flowgraph. 


The descriptors are rigidly set forth for the user in the data sets 
supplied for each TAM. These data sets are a presentation of the TAM data in 
the proper format for SAINT program processing. Data sets for each leg of 
the RTP are supplied to the user in Appendix 0. The data sets for each of 
the 19 modules which comprise the RTP are presented in Appendix D under the 
same MODULE .DAT. 


There are task descriptors within all these data which are subject 
to change by the user. They are: 


(a) Task essentiality 
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Figure 3-16 Typical RTP Flowgraph (Leg 4) 
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Figure 3-17 RTP Leg 4 (1 of 2) 
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Figure 3-17 RTP Leg 4 (2 of 2) 
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task number 
number of releases required for first release of task N 


number of releases required for subsequent release of task N 


time descriptors: 


ty parameter set number 
ty distribution type 
task type: 

S single operator task 
joint task 

E either task 

Q equipment task 

C cyclic task 

F gap filler task 


task essentially (0 < E < 1) 

task class 

task adjustment factor 

task involved in data collection such as a mark node (S = M) ora 


statistics node (S = F, A, B, I, or D) 


Figure 3-18 SAINT Flowgraph Descriptors (Copied from SAINT, Vol. 1) 


(b) Values contained in the parameter set, NOT the parameter set number 
(c) Distribution type 

(d) Operator characteristics 

(e) Time allotted to mission 


Task essentiality is a number between 0.00 and 1.00 which indicates 
the importance of a task to a mission. If the mission is estimated by 
SAINT's benchmark iterations to run over the time allotted, tasks with the 
lower essentialities will be skipped, while tasks with an essentiality of 1.0 
will always be performed. Parameter set values (i.e., mean, minimum and 
maximum times allowed for the completion of the task) will probably need to 
be changed with every engineering design change. Some parameter set values 
are supplied for the evaluation, cataloged (i.e., manual access, visual ac- 
cess) in the form of an index called PARAMINDX. DAT (Appendix G). The user 
will assign these values to the parameter set identification number already 
designated to the task. The distribution type is a code number identifying 


one of the 11 available types (paragraph 2.2.3.2.1) of time distributions in- 
to which the parameter set will fit. 


Operator characteristics (@.g., speed, accuracy, stress threshold, 
and allotted time) are the final values subject to change by the evaluators. 


3.3.6.3.1 Parameter Set Interchangeability. Each task has been assigned a 
parameter set identification number identical to the task identification num- 
ber. These numbers range from 1 to 135. Parameter sets which contain para- 
meters equal to zero have been numbered 200 to 250. Because the highest task 
number is 135, the user can quickly identify a parameter set numbered in the 
200-series as a zero value parameter set able to be assigned to any task. 


A task which has been assigned a zero value parameter set (200 to 
250) is distinquished as a dormant task. This is the feature which allows 
the ITD to contain several methods of performing the same instructor task. 
The user need only identify which method applies to the IOS under study as 
active, and set the others dormant. The model processes the dormant tasks 
but they will not have an effect on the outcome of the mission, 


SAINT can be used to change parameter sets by preprogrammed in- 
structions. This feature allows the modular construction of the RTP to work. 
The 19 modules comprising the RTP are often repeated within each leg of the 
RTP. Sometimes in this repetition the user may want a dormant task to become 
active, or vice-versa. SAINT has capability to allow the completion of a de- 
signated task to trigger parameter set changes. That is, it can remove para- 
meter sets by their identification number and replace them with another. Be- 
cause SAINT removes parameter sets by their identification numbers, and not 
by the task they are assigned to, it was imperative to number each parameter 
set distinctly although the values in some sets may be equal (see User's 
Guide, Appendix I, for modification procedure). 
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3.3.6.4 Model Parameter Control. SAINT's parameter controls make possible a 
clear, concise, and simple method of keeping track of how parameter changes 
affect the model of a simulation. 


Task parameter sets may be changed as the user sees fit. SAINT's 
output keeps track of these changes by providing a data echo check. This 
echo check sends back the data the user has specified in a labelled format, 
following the same data progression as the SAINT input specification with 
appropriate headings and explanations, 


In the data echo check, the parameter set assigned to a task, the 
task type, the distribution type, and the assigned factors are labelled and 
shown (Figure 3-19). Any specified parameter set substitutions and the task 
which triggers such substitutions are also presented in the echo check. In 
the printout of the detailed iterations, both the time of replacement, the 
parameter set replaced and the new parameter set are flagged and printed 
(Figures 3-20). The length of time for which each operator was busy or idle 
is shown in the summary of iteration along with the maximum and minimum 
stress on each operator and the tasks on which these stress values occurred 
(Figure 3-21). The task summary report supplies the user with the number of 
times a task is released, failed, or skipped. The mean, minimum, maximum and 
standard of stress values on each task are also presented in this summary, as 
are the mean, minimum, maximum and standard of times tasks are realized and 
completed (Figure 3-22). 


A task which holds a particular interest for the user can be desig- 
nated a statistics task (see User's Guide, Appendix I) and will have a histo- 
gram aiong with a set of time statistics printed out for that task. 


Use can be made of some of SAINT's categorizations to determine sta- 
tistics of interest that are not calculated as such by SAINT. For example, 
if all manual access tasks have been designated as task type 2, and the user 
is interested in the total amount of time spent manually accessing devices, 
he/she need only sum the times of all type 2 tasks. If this becomes a common 
necessary parameter, the software can be adopted to calculate this particular 
value, or any other value of specific interest. 


3.3.6.5 Model Outputs. SAINT's output is divided into benchmark iterations 
and run iterations, 


3.3.6.5.1 Benchmark Iterations. The benchmark iteration output contains 
three sections: 


(a) Echo Check of Input Data. See paragraph 3.3.6.4 and Figure 3-19. 
Portions of the input are detailed following the same data progres- 
sion as the SAINT input specification with appropriate headings and 
explanations. 
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Echo Check (3 of 5) 


Figure 3-19 
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Figure 3-19 Echo Check (4 of 5) 
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Figure 3-19 Echo Check (5 of 5) 
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(b) Detailed Output of Benchmark Iteration. Benchmark iterations (Fig- 
ure 3-23) do not take operator characteristics into account during 
the calculations; thus, no stress values are produced. The data 
put forth are task number, type and essentiality, start and expec- 
ted end time, regular probability, and modified probability. An 


asterisk in the appropriate column shows the operator(s) assigned 
to that task. 
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(c) Benchmark Iteration Results. These results (Figure 3-24) show the 
time remaining statistics for each operator as each assigned task 
is performed. SAINT applies the operator characteristics to these 
statistics in performing the detailed run iterations. Every operator 
performed task (i.e., not equipment tasks) must be performed in the 


benchmark iteration, or else, due to SAINT’s structure, the detailed 
iterations will not be performed. 


3.3.6.5.2 Run Iterations. The run iteration output is in four sections: 


(a) Detailed Iteration Output. This output (see Figure 3-20) is in the 
same form as the benchmark iterations. The difference lies in the 
fact that operator characteristics are considered in the detailed 
iterations; thus, stress values on the operators are given. 


(b) Summary of Iteration. This summary (see Figure 3-21) is given 
after each iteration. It provides information on each operator's 


performance times (idle time, busy time, time available, total 
time) and stress for that iteration. 


(c) Task Summary Report. This report (see Figure 3-22) gives the fol- 
lowing statistics on each task: 


(1) Mean, minimum, maximum times and standard deviations on the 
time the task is realized and the time the task is completed. 


(2) Task stress (Cohesiveness is also presented, but this value 
was never used in this study. Cohesiveness represents the 
amount of cooperation between operators. Because all opera- 


tors are segments of one human being, this parameter cannot be 
applied meaningfully). 
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(3) Number of task releases, failures and skips. 
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(d) Mission Summary Report. This report (see Figure 3-25) gives the 
times consumed by the operators broken down according to busy, 
idle, and total times. The stresses felt by the operators during 
the mission are also presented here. The final output of this re- 


port contains the number of successful and unsuccessful missions 
performed. 


woe 


ee pce ae eas ee nel cen a a Mi 
. been Sage 2 
ay Cees 5 “ & 
x > a. Ret Rat nn hae 


78 


| 
| 
| 


ES: ve SRT RPS SAR TOP peaKEee PPE TATRA eae os Ri TS 


ee ELST ERAS SRO NRT FAERIE 


a v 
a 
a 5) 
‘7 
‘ oc "0 00 0 CO 6 00 0 00'°0 00 0 OO O# 00° 00 0 000 000 oo 0 coc ool t O00 Le OBZ O06 'G6Zt Of YNIS tb 
00 °0 00°0 CO’O O00 O 00 oe Q0-OwxO0 O 00° 00 O* O00 'O co a ooc *t O0C I $B bist 06 tGZt Ot ONIS Cr 
oc 0 00 O COC OC O 00 * 00 0 000 OO 0 He 00 G oc 0 co GO ooc 1 000 I QO EeZLdt C6 6SZT OF IMS GS 
" oe 0 00 0 CO 0 00 0 “Q219s9oes 2H @wo0 0 00 O» oc 0 co 0 ooa “tf 000 ¢ CG& tot e€ Geet OF LK cy 
’ oc 0 00 0 CO Cc 00 0 ‘O# OO OF GO O* OO OF OO OF O00 Os QO O# OO 0 coe ooc 1 OOC 1 ce Ce“t cE O€¢t OT 1 M8 99 
4 —7— 00°0°09°O CO "O~00"9 $0 9 00°O 00°'O 00 °0°° 00°0-- 00°0"~ 00“9R- -OG"G"-—CO"O- ~00G7T OOC "I ~"-E9*zyyl —~“42'Ocet ‘OT SONI co" 
7 000 00 0 C0 'G 00'0 00 0 000 00°O0 00°90 000 0090 00 '0# OO 9 co 0 009‘! O0C ¢ é2 'BCEl ct @det Of ONIS c? 
--— 00.0 00'0 CO © 00:0 00 0 009 00.0 _00 '0_.00'0.. 00:0 90 08. 00.0... CO O_ 00071000 I_ Cl. foc! 46 Blt OF Onis Oo? _ 
iS 00 0 000 CO 06 000 00'0 900 0 000 GO'0 09 9 9000 O08 O# C090 co Oo 000 ft OOO Jt 26 CEUT OS OFIT OF OATS &S 
‘ oe 0 90 0 COO 00'0 00 0 000 00 O 00'0* COD 000 000 oc O co Oo eoc 1 000 ¢ eF Ec4t éc OEIL OF OSNIS Ss? 
an oo 2 09 '0 CO 0 00000 0 0090 900 04 000 000 000 00:0: -00 0-00 @ ~-900'!t -000 1 cE CELI vc OETT 9 fF ONTS wo 
7 00 °0 00 ‘0 00 '0 00 0 00'0 000 900'°0 00'0 0090 000 00 O* 00 0 co oO 00c ‘tT 000 1 aq OPtt 62 VETt Ot Onis Qs 
‘ 00 0 00 '0 CO'O 00'0 00°70 06 79 900 0 00°70 00 O0* 00 O* 00 0 00.0 co 0 00G°f 000 1 €? Cest 6c OCTT Of JTAW ve “ 
Ay “OG ‘0 -O0 ‘0° CO"O-O0"°0" 00°08 ‘00 &~00 ‘0* 00 “0¥ 00" 0#- 00 “O# (00 “Of ~OO"Or™ COO OORT OOO "I~ =~ cOCETS- SC GENT OF Lan tS" = 
:" s oc ‘0 00 0 CO'0 00'0 00 0 0090 000 O00 'O* 00 O# 00 O# 00 O* O00 0 coo OS6 O OSE O éc Cet et rtrt O fF sey > Co 
c, 000 00 0 CO'O 00:0 00 O* 00 ‘O# OO O* 00 'O# OO O# 00 O# 00 O# OO O... CO.G _000 I GOO 5. by Tits t@ vaolt Os Linw cot — 
z 00 0 00°70 CO 0 00 0 00 C 0090 000 900°O# 000 0090 OO O# OC O co 0 ooe {t 000 It Ef C66 68 SSE ot 11nw TOL ww 
n i 00 '0 000 COO 00'0 00 0 G00 000 OO O# 00 O# 00 O* 00 O# OC O co 0 0oO $ 000 I é0 S96 68 S9& 9s sims oot o 
f : 00°0 000 CO G-00-0 00 0 -00°0 CO 0* 00 0 -:00 O 00 0--00 0- - OC O-- GO O-— OOO 1 OCC I ‘t6 9BO! - 68 S96 Oot ONIS Q6 . 
en : 00 (0 00 0 CO'O 900 '0 00 “0* 900 0 00 O 00'O O00 : 00 0 00 'O oc 0 ca 0 ooo I O0C $f ct sgot 6B S9é& Or sis 66 f3) 
di 00 '0 00 '0 CO'O 00'°0 00 0 000 00 0 00'0# 00.90 00°70 00 O# 00 0 co 0 000 1 000.1 £8 S96 68 $9b QO 5 LIN £4 » 
Rec j ——"-—00°0"00~0 -CO"O- 000-000 ~ 00-0 00 ‘O° -00“0*8~00"OF- 00° 0¥-00"Or— Sno. ——CO “O- ~0007T COO"T ~ LB SSE ~ FE“TO’ ocr Lay 96 — 
‘? 000 GOO COO 00 0 00 0* OO O* 00 O* 00 'O# O00 ‘O« Be “O* OO Of ie] co oO 000"! QO0O0 1 FE 708 SE 906 or 11ns S6 
2 , .——. 00 0 OC O CO‘:O 00 ‘0.00 0*.00 O* OO O# 00 O* 00 Ox 00 Ox_00..0« 86 Q_ ca 8 __d08. 1000 l.. FE‘SO0&. _ SE 904. O.7 1 IN eB x 
; 000 00'0 COG 900'0 00 0 0090 90090 GO O# CO O# 80 ‘Ox 00 O* OC 0 (ovemme) oo. 1 000 I éS ‘806 Ee SOpv Os JINN 9B ca 
\ 00 ‘0 00 ‘0 00 '0 00:0 0070 000 00 O O00 O# 00 ‘0* 00 O* 00 Ox 00 0 co a ooo 1 090 1 c€ Sb €€ Sev Ob 110s $e co 
1 00 0 00 °0 COC 000 000 00°0 900 00°O# 00 O* 00 O*# 00 O08 - OC-0--— CO O --O00'! COC ft ce Ger cE S@b Or iL InW te f= 
00 0 00 0 00 0 00'°0 000 000 00'O0 00 O* 00 O* 00 O* 00 O* 00 0 co 'O aoc "ft O00 1t cc SOb €& Seb Ot s1nW ce = 
00 °0_00 '0 CO "0 _00_0 00 O* GO O* 00 '0* 00 0 00°0 000 000 0600 60 6 000 3 000 t FE 906 __ €€ 68 Ob LINK 28 oO 
0G"O 0O"D CO"C- OS" O- 00-0 00 0 000" 00 O¥ 0D OX 00 OF 00-OF “00-0 COS 000 T OOC’T Ce Seb “te Sev or IAN “3° c 
00°0 00 °0 CO 0 00:0 00:0 000 000 00°O# 00 O# 00 O* 00 O# O00 0 co 0 000 Tt 000 1 EC S6b ce Ser Or Lams UB @ 
000 0C ‘0 CO'0 000 0070 00 0 00'0 OO O* OO OR 0O OX 00 Of. OO .O CO O.. O00 1.000-1 ce $Bb te 19% Oot LW &L CO 
00 0 00 0 00 6 900 0 000 00 0 000 OO O* 00 Ox OO O* 00 O# OC O co 0 ooo Tt O00 I to Sb te (9b Oot Liaw BZ 
00°O 00 0 CO O 00 O 00 O# OO O# 00 O*# OO O# OO O# 00 0% 00 O# O00 0 co 0 000_T 000 It te 9b Te 9b Qt 11NWw tZ 
ros eeentTo (9b--SWEL £Y 9@ --23S UBLBWYMWd AG ABQV1deN 612 LBS MALAIViWdeene o> 
weaele 19h ana 1v 84 13S eee ta AM G3IV7d3YN CIS L355 BBLBNVUVdw eas Ww 
000 00 0 CO 0 00 0.00 0 00'°0 000 00 0# 00 0* 00 O» 00 O# oo co oOo : goc..t 000 Tt le t9b bt Sot OF snes rad 1 
——"00"0 06 “O CO “O00 000 “08% “00 “OOO “OF “OOO OO oe 08 “O* OO OF o8 = GOAT OOO "Tt" O0O~t bT-GSt BISSb” OTT 1TH ay ~ 
00 0 00 0 00'0 00 0 00 0 000 9000 O00 O* 00 O* 00 Ox 00 co Oo ooo ft 000 I bl SGb til SGy Ot sw Sit 
- 00°9 00'0 CO0'6 00°0 000 00°60 00 0 OO OF OO ‘Ox OO “Ox 00 On 88 8 --CO.0 —.000 1-000 I vt Sb €& cup Ot LAM ott @ 
000 00 0 C0 0 06'°0 00 0 0090 000 O00 O* 00 O* 0G 0» 00 Oe 00 0 co Oo ooc I 00° | Cf chb Of CEbd ofr AINW eu a 
00 °0 00 °0 0CO'0 006 0 00 0 000 900 0 O00 O#* OO O# OO Ow OO O# GOO co Oo ooe 1 000 I ce C€t cO Cer Oot 1 Wits crt Ss 
- 00°0 000 COO OC O 00 0 00 O* OO O* 00 O# OO O# OO O*-00 O* OC O co c---00C0 I O00 I co tet 20 Ee Or sts uit ™ 
00 0 00 '°0 00 6 00 0 00 0 000 00 0 OO O* 00 O« Bo O# 00 sa oe O co GO OSé 0 OSE DO co Leb Rb Ocby Oy 1 wws t = 
00 0 00 0 00 ‘GO 00 '0 00 0* 00 O* 00 0# 00 O* 00 O# 00 O* 00 O* OC 0 co 0 oea § 000 Tt fb Ceb Fb OC Oot Imn408 _ 68 i 
—"IO U OF O CO-O 00 U 00 GO 00 0 “0G°O “D0 OF DD D* COOK O0 O# ~ 000 DOG DON" T O00 TF COG 00 0 OT «J TnW 98 
oe 0 000 COO 00'°0 00 0 000 900 O00 O# 09 9% 00 O# OO OF OC O oo Oo ooe 1 00c IT co a 00 0 Of sw $B 
- 00°O0 00 °0 CO G.00:0.00 0 000 00 0 OO O# 00 'O# OO Ox 00 Of OO O-~ COO- OOA'I OOOT - COD 00 0 Ot LIMe te 
00 0 00 '0 CO 6 00:0 000 000 000 O00 O* 00 O* OO O*# 00 O* OC O co Oo ooc 1 00C 1 co oO 00 0 Or Line €a 
00 ‘O 00 ‘0 00 'O 00'0 00 'O* 00 O* 00 O* CO O 00 0 00 0 00 0 oo 0 co Oo ooo | 000 It 9b Ce 00 0 Oo fF LAW eo 
-- 00°0 00‘°0 CO0“GQ-00°0' 00 0 000 000 OO O0#:00 O* 00 O* 00 O# -00 O-—-CO COC --O0C 1 O00 f--- COO 00 0 or rays 18 
00 °0 00 0 00 0 00 0 oa oO 009 00 0 OO O# 00 O« OO O# OO O# OC O co 0 cog t G0C t coo 00 0 os Lan ob 
00'0 00 '0 CO '0 00'0 000 00 0 000 00 O* 00 0* 00 0» 00 O* 08 0 COO O00 tf OOC It COG _ 90 O _ OF LIAN OL. 
—<""90°0° 00°O 00 “TUG OU" oo” Oo” 00 °O~ 00°O~ OO OX UO -O*" 00 OF OO OF” OOO COS "000 FT OOO 'T Go-Go 00 0 Oo 1” J70W BZ 
oo 0 000 CO 'O 00°0 OO O# OO O* 00 O* OO O# OO O* 00 0# 00 OF OC O co Oo ooo ft O00 1 co oe 00 0 Oot 1 1ne Lh 
; i tT ot ) 8 Zz 7 S b & 2 1 SS3Su1S S309 qovd dOud AWIL ANZ TW SS3 SdAl Y2BaNA 
F ee) See fee me See oe tee: eee ee ey SamlwaA SS3Y1S BOLVYSdD ~- - - - - - - = -- = WSVL att 934u O31L93dx3 Levis MSYL SVL YSVL 
I 


1 S38WAN NOILVUGLI WUVWISINSES JO LAdLNG_G3tIvL3ad 


¥ " 


AER QTE PS SESE EGET RTE ae, Rey Xo eR SS TE 


$24 


L£2°S2E 8B IZE 00 0OT FO'r SS vee 00 0 coo  o7 oor 000 ooo sB'se IB te GO Gor 4420 v6 €2 9 ab 
EP Ze = FSS QD ONT «LO ‘Tere ahtoitear 90 O : 000 8660 G be ‘eSst SE vest GO-OOL OS I €6 Zest S Bb 
1S. OG'0——C GOTst4Z'G SF OOO OO 00G COG 96 '86Zt ES OvZI GOGO! Of O1 cB Tzer ob 8b 
BZ 6L42 Eb'ELf 00 OOF 111 sriINGUIINa GWO f° 000 GOG £1°996 6€ 606 COGOI BE OT €8 6E6 € Bb 
SL 442 €6‘EL2_ 00 00% 10 't 2F 942 00 ¢ GOG  OG'0GT 000 GOOG 241.996 6€ 606 CO GOI G2 Oi €6 6E6 2 Gb | 
“Ea"a-* —0G'O 00" 00T 19-95 “0 “> 00°00" G 0G BGT —00-0- GO "GSE ESOL GT SES—CO-COT SGOT BT 8aGt— 17) BE 7 
SZ 000, _.00 OOTL._IZ0.. S70 -_. 00_0__ -.c0 :0___.002003....000 ___. CO 0. 9 L16L61_06 0261 CO.COL_LTOL. _te.iset ob Lb J 
BC'29 26°65 00001 S£°0 8b0F 000 COG oCoot 00:0 COG oO StI! 1esscol Go cor tC or “4t Bser ot 9b A 
Z2e‘see 24:82 OO‘OOS SE't %S Zeb 00 0 coo = OD} GT 000s (ts CES «CIE HEE COCO SE Or seBEte CE Sb nae 
ce See 2 'eév 00 Oot Se't tS deb 0090 SO OG —OG'0GT_00'0 = GO GE ES TI DES CO COT 6E 01 4B Ese 2 Se = 
Te EBI EF '9ZT O0'OOT ZE I GE OT 00 0 CoG OF OOF 00:0 COG 49 °S4Et 10 BIST CO'GOT 11 t SO 241 ¢ by rd 
esz9  20'6S COCO EL 8b 6F 00°06 06 6 ~ 68 'OOTO5°O"” GO DO «=F -SLTT Be STITT 66 GO’ GG oT ez -avtr it eb co ; 
G2'S0S BP'ICS O0'0O!- ¥6'O - ES'POS 00 0- GO'O — 0G :0G1—00 0——-GO'G—-—0E‘0S- -S1-9» GOGO! 460 808 £ 2b = : 
69‘S0S BF'IGS OOOO! £6'G ES PCS 00°90 GOO OF DUT 00:0 COC O0F 0S SIG GO GOI 460 80 By 9 ob aS . 
TeE8l ESF'°9Zt 00 O01 ce "t SE O81 000 GO'o 0G:0Gr 000 3 an= LE_OGPI bb T6St GO ‘COI €6 1 BI 96st S 2b cS 
“1G~a ~~ 0070 ~—-00-00T-—TZ-G — ~S5"0" ~— 00-0 GO-g —— OF -OGt— 00-04. ZB 200 e EB ERST OC COT Ee Or be GZt eee = : 
ZESEy 12 "Rey OO ‘OOT FE 'T bS Zeb 00°0 Gog 9g ‘0G 90 0 ‘G 66 EIOT 6S 266 GO COL be'Oi 0 886 —& é& ; 
GE‘SEb 12 ‘3cv__00 O0T...bE11... SS "2h 00.0._..G0-¢ __aG-0G1__ 00-60 2G__._44.,E101_ 66. 266_. GO-COt_._ bb-O1. _.€0 886_..-2. dv T 
BS ‘29 26 6S 00 COI SiG” sb OF 00°0 60-6 — 0G OG! 00 0 GOS LF Balt Se SEtT GO OF BIOL ~ ae Batt eb 2 R 
16% OOD =O '00T 14°09 $90 =—00 0 GOO --- OO '00T - 00 ‘0-— - CO 'G- —- EE "SSGIT SZ ‘SITt CO-¢ <@8 Be cert »& 9 = : 
B 42 gb 'Ece 90 ‘00f 20°F 27 '94z 90:0 800 001001 90:0 oOG g2'seg 18 EBr GOdul $7 8 Sb TOS & 9 
2 °EL2 26 E22 00 ‘00t_ 20: 27922 __00'0 Coo oa oct 00 - Co: ge Gea (8 EBb CO Got $7 8 Sb “tog e 9 2 
£6 $< §6-54=—08-Got—¥9-u-—-— Se" 00-0 00-06-08 I 00-0 00-20 BEI CO COT BEB 89 TLS S = ; 
152, 99:0, OO100t 12:0 SF 9, 99.9 . GO G__ 09001 00.0. -GO :0...EB.8ES.-40 LOS GO."GOT . G5 9 £2 QI b S e 
82°42 26 'ELZ 00001 E0%t 29 ‘942 00 °0 60'G¢  OoG*uot 00 0 ~ GO: ef aor Zt 22° GOGO! @6'9 Ee SE € § — 
BL ete 26 ELE 90 '00T ECT 27922 00'0 CoS OF oor 00° Cou Eset Zt ec COvot BO9 ee 98 @ S 2 ; 
e@ 2 00 ‘COT-- T9-G = - 9S 0-000: -60 'G- ~--00“0GT-—00-0 --— G0-G——~-SS-8Lb Ga bby —GO -GOT--b£ 9 S€ asr-- 1 S - = 
1S'% OOO ~~ CO OOT_—t4 0 S970 000 CO'O  —oO'00T_ 000 00 OE 'SZZt BH STLZI CO“COT OF Of 06 Zbct_ eb b co 
“BL ELE e6 ELE O0-OOT 207 F —"e9"9zz~" 0-0 GO-GO TOGO Gg be "ebS “e090 "GOT Ee Or TS ETE Eb 
GL‘6le 26 £42 00°00 ZO'r 27 942 000 CO'O = =6oc‘ocr 00: be? S cO'9ee COCO! EZ Or Sle @ b 
Ez ° 000 G0 OOT_.T9'O . 96° 00° . 606 ...00-001 ..00'O.. SOO. Ee-HeOt29 Ize Go-cot OGor G2 ocr ¥ Bas ee 
6t'y 00°O O0'001 £4°6 Trt 00 ‘0 gO';C OG OCT 00'°0 CO; g2a'Bb9e 9 9662 GO ‘COI 48 21 42042 »& £ ) { 
£9969 T2'627 00001 Le't- IG'EER? 00°0 -GOO -00 0OI--00'0 - - G0 'O-—-62 ‘StI -BE GZET CO‘OOI GB EI se corr E E€ o % 
£9 SEP 18°29 O0'OOT Zb't 10 EE9 00°0 COC OC OCr 000 GOO 42 Bb BE-ZZEI CO'GOI 268 21 SE 20b1 2 € 
_56SbE__29 BEE 00°00! 6S 'I 26 2eE_00'0 GO G 0G 0GT_00'0_ | GOO. PG ETSI TZ O4bi CO CO! IF St 22 post. 1 € v 
EB‘IOt bese Ad‘OOr E'S 2246 080 COO OOOOT COO COO Sz bIBe 26 B242 CO‘OCO1 OLE Bs oe PY 2 > 
EE LZSL 94 °S¥L.00 OO! -29'3. Ce "ESL 00°0 ~ OO°O... O0.0GI ..00 ‘0-00 ‘O--...08 BEST Ce SICI CO‘GOr Or b1 eB arcr €. =2- Dm { 
E6262 9¥"éb£ O00‘00f é9°% Sb €sZ 00°0 GO..0 OQ 00T 000 COC OB 'SEST SEEISI CO COI Of bt eBarst Ff 2@ — 
B0'Sth ES SEF OO ‘OOT Té't £5 6b 00 ‘0 00°O — oO ODT 00 O cod ve eect BI Ev9t COOOL 47 EF CO Sat Tt z ra 
EB‘IOL ¥E'S’ OO'OOT CE ‘F 22:46 00°09 GO'0 O09 '00r 00°0 coo 96 ‘6582 9b '29¢2 COCO! bE BI eb bise + I 
EC 4S4__ Fe "642 0000! 69 ‘5 Sv ‘&S4_.00'0__ G06 0G 00t_00'0_ GO bE GbSl Eb ‘bSST GOGO! SC 61 49 20cT_ E i 
E: “GE 252 2¥ Bel O0 OOF 7 Sb ESZ—000 GO 0 0UT— 00-0 FO o— bE GBST Eb BSST GO GOr~BU-&I~ 29" a0 e i 
be & GO'S#h ES ‘SEP OOOO! Té I ES ‘6€b 00°0 Goo oC'ocr 00'0 380° 6S ‘OLLI 24 9451 GOGO! SS'8t da eect 1 1 3 
“XVH ‘NIW "S80 ‘ON “ABd ‘dis NvaW “Xv” NIW "S00 GN Ba TiS WS "XW NIM “SOG ON ‘AB0'GLS NvBN = dO BON 
f meccce-ONINIVHSY BWIL ONTLIVM-----=-- 0 ---=- ONINIVIGY SHIL WILNHSSS-NON-----  --=----ONINIVISY BHIL WILN3ESS3----+-- 
ges _ we  SEINSBU_NOLLVMSLT WOVNRONEA OO sha a Rte ef hg hoe : 
| 
ESA ares pepretaqeres TS : o angunader mesa Bed asin cairns enh hs sw eyes uk, anahe = 


RTS mperes = z ‘ 
Se a Gea SS Sey Te te he ie PES Hive = wen ese n 


TAN Aes a 


gap 


acc 6 ce 
” * 
7} 


Goak ooo 


tt nd os ot td 


ANxHakKKOO 
ooanea0 


ee + 
nt ont oad wt ont mt ont 


wWAMUIONdOS 
ecoooscse 


ododdods 
nuuaess 


wt ott ed ne 


. =2-----STRESS---- = 
MEAN STD. MIN MAX. 


MAX 
98. 
98. 
98. 
98. 
98, 
98 
98 


POR SCOURS! 


aise 


ivlvlvlvlviviw] 
NON AAO 


MIN 
8g 
83 
83. 
83 
Bs. 
83 
83 


oa 
STetvivistaty 
SFOSKOSOHOOS 
errerere ; 
cococsco 


tv rte ot tt 


TIE USED ---~e6 


Le PEAG RIAD Ce oY 


STD 


AMSA 


balalabebeied 
TESST OS 
ielulelulul yyy] 
MHMMAMMOM* 


~-2-~" TOTAL 
MEAN 


ux f 
rs j 
ag 
t LSEH0LVCO 2 = { 
i u aw ] 
' (6 Neteoood = ! 
tL YDOGRrWn Q = { 
&- 1 envgean 4 | 
Se et CIOL AN 
ot j fou 
& to yam. oe 
wot as oe dieters | 2p re 
© wZ Yocom ss 5 & 
Se YRARGAN w 
> =E Ynmarsco 48 ui x 
Ge see SG OG 
x Ge : A 
BW Caners 3B 33 
x . . Se | as 
2 SO rocneda 5 Zz { 
Nome S28 NGO A SF : 
2 Lm 69) —— es H x 
‘ a ‘ 
cr N ota 3 
mol. 7¢xsuco , , 
O 12 Kooneed y 
= tt NOOKQa~ ‘ 
at Q Sannees ' 


1 
2 
2 
2 
2 

tides 


as 


f goon-oo 3 
rx ANKO-na 3 
14 =M-000 4 
14 aac oTe a: 
H ell WLM} Se 
F ] 
1 soon yw 3 
wz -odaN-— 4 
Le Near ow a 
HE NaH ONO Fi 
re mee eC # 
3 onnd999 3 
1 BQ Nemes : a 
j Cm Gwe 7, 
i 1 me t Prd 
Hy Merstoonn 4a 
tz é a 
ig Sgoan ye M 
tQ UeeegARn % 
fo TeaHeoo 
: erates] 


Se00S500 “4 
yo GQCaGeS 4 
he <x ° it 
=J S0oc0C000 4 
ee SOOoC900 5 

<x veetcrds r 

‘ > AMMAN dQ 

‘ag 


ANOPTAGN 


PERATOR 
NUNGER 


ect er orarrenaiee 


Figure 3-25 Mission Summary Report 
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3.3.6.6 User Considerations. Descriptive terms have been put into the model ¥ 
and the evaluative terms have been supplied by SAINT. The evaluative terms 4 
must now be analyzed. 4 

The first section of the output is an echo check of the data (para- 3 


of the data that have been input. Although there is no need for interpreta- 
tion, these data should be checked to ensure that the descriptive terms which 
have been supplied are actually those which were desired. 


The detailed output of benchmark iteration is open to interpretation 
by the user/evaluator. All through this interpretation it must be kept in 
mind that the benchmarks do not take any of the operator char o>ristics into 
account, The benchmarks supply the start time of each task, t, xpected end 
time and the modified probability. Any other data presented are merely a re- 
presentation of the task descriptors input. The expected end time is calcu- 
lated from the parameter set and the distribution type which describes the 
task. The start time of a task is dictated by the sequence of the tasks and 
the end time of the predecessor task. 
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graph 3.3.6.5) and requires no interpretation. It is merely a representation 


The format for the result presentation of the detailed iteration is A 
identical to that of the benchmark iterations. In the detailed iteration, “ 
however, the operator characteristics have an effect on the results. The re- 


sults set forth in the detailed iterations and affected by the operator char- 
acteristics are as follows: 


es 


seers Tah EARLE et 


(a) Task Start Time. The time, estimated in the benchmark iterations, 
at which the operator(s) assigned to the task will have finished 
the preceding task and can key in the new task. 


wy? 


(b) Task End Time. The time at which the task is expected to be com- 
pleted. This depends on the time at which the task was started, 
and the expected duration of the task. The duration of the task 


is, in turn, dependent upon the values contained in the parameter 
set and the distribution type chosen. 


ae 
Panne 


(c) Operator Stress. A time pressure imposed on an operator by a dis- 
crepancy between the amount of work to be done and the time remain- 
ing for doing it. Prior to the start of a task, the status of each 
operator assigned to the task is assessed by SAINT. The time which 
the task will consume is subtracted from the time available, and 


the change in the stress of the operator caused by this task is 
calculated. 


ig ancl Sentient e tinea 
. st ‘ a 6 % 23. 
7 ae es fete oe 
BES ase ROR 5 RRB A a al ANS. Srentee a te Jae! 


: ate 2k 
ae a ees 


(d;} Task Stress. The cumulative effect of the operators assigned to a 
task upon each other. One operator under great stress will in- 


crease the stress upon the other operators assigned to the same 
task. 
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3.3.6.6 User Considerations (Cont'd ‘ 

The iteration summary provides information on each operator's per- “s 

formance time in terms of available, busy, idle and used times. The stress " 

statistics set forth are the mean stress, standard deviation, minimum and 4 

maximum stresses and the tasks in which the extreme stress values occurred. a 

These two sets of statistics (detailed iteration, iteration summary) are the “4 

main body of data from which evaluations may be made. = 

a 

a The interpretation of the, stress values is the most important step 4 
Be. in an evaluation. Siegel and Wolf portray any stress value up to and “4 
Ee including the operator's stress threshold as an organizing influence which 3 


causes a decrease in the mean time of task completion, and an increase in the 
probability of the success of the task. Alternatively, a stress on the oper- 
ator which is beyond the threshold causes a disorganizing effect (i.e., the 
mean task completion time increases and the probability of successfully com- 
pleting the task decreases). Thus, when an evaluator discovers that the imple- 
mentation of a particular device on an existing IOS lowers the average stress 
on the operators, it is clear that the device is beneficial. Correspondingly, 
if the evaluator finds that the design evaluation shows the operators' stress 
values to exceed their stress thresholds, the design should be altered. 
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The next useful output is the Task Summary Report which contains 
five groups of statistics (see also Figure 3-27): 
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1 (e) Counters 
a (f) Time task released 
Al (g) Time task completed | 
‘ (h) Task stress 
: (i) Task cohesiveness. 


The counters group contains the number of times each task was re- 
leased, skipped or failed. The other groups contain the mean, standard devi- 
ation, minimum and maximum values for the times, stress and cohesiveness. 


The Mission Summary Report allows the evaluator/user to learn exact- 
ly how much of each operator's time was used productively or idled, along 
with the stress values they attained throughout the mission. 
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1 The number of successful missions per one hundred is also contained 
: in this summary. 
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> SIEGEL, A., WOLF, J., et al. Modification of the Siegel-Wolf Operator 
Simulation Model for On-Line Experimentation. Aerospace Medical Research 


Laboratory, Wright-Patterson Air Force Base, Ohio, June 1971. 
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The final output is a histogram of any user-designated task. If a 
particular task is under question, it may be designated a statistics task and 


the evaluator may obtain further time data from the histogram (see User's 
Guide, Appendix I). 


The user can now run the RTP with data from several different I0Ss 
and compare the results. If different I0S activities are used, the tasks in 
which those data are used can be marked for a histogram. If all the other 
data are kept constant in both parts of the comparison, the effect of the dif- 
ferent device on the mission can be readily noted. The number of successful 
missions may differ. The different devices may cause variations in the 
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stresses throughout the mission. The average stress may vary or remain the 
same from one station to another. If the average stress varies (Mission Sum- 
mary Report), the comparison then becomes obvious. If the average stress | 
does not vary however, there may be differences in the time of occurrence of 1 
the stress (detailed iteration), Times of completion of both the entire mis- 
sion and any tasks in question supply the easiest method for direct compari- 
son. 
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3.3.7 Validation of Objective Methodology | 3 


3.3.7.1 Validation of Model Behavior. A validation of the evaluation metho- 
dology was carried out in order to determine the sensitivity of the model to 
A the data which will be used on I0S design analysis. 


In order to examine the model's sensitivity to changes in interface 
be methods, the keyboard TAM was run twice, differing the active and dormant 

BB tasks each time. The first time the keyboard TAM was run, the tasks were 
marked active in a manner which indicates that the instructor checks the page 
3 called up on the CRT before pressing the insert button. This is shown in 

a Figure 3-26 under the checklist column EFFECT SIM. In the second run it was 
a assumed that the display being called would not directly affect the simula- 
4 tion (i.e., an assessment task), and the instructor weuld not check the entry 
E: before keying INSERT. The tasks checked active or dormant are in the NO EF- 
a FECT column on Figure 3-26. 
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Th The parameters resulting from the keyboard (EFFECT SIM) TAM were 

a then assigned to each keyboard task in leg 4 of the RTP. The operators were 
assigned average characteristics and an estimated feasible time for completion 
of the mission. This wi.i be referred to as the standard mission throughout 
the remainder of this report. 
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Next, the parameters resulting from the keyboard (NO EFFECT) TAM re- 
placed the previous keyboard tasks while all other parameters remained the 
same, 


The standard mission was run again, this time to determine the ef- 
He fect of changing the speed factor of the operators. In another run of the 
i standard mission, the stress thresholds of the operators were set below aver- 
iL age (i.e., less than 2.5). 
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KEYBOARD TAM 18 4 

Effect. Sim. No Effect “eg 
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1 Attention-Getter mine 4 

10 Visually Access Displa i 

ually play wae 4 

40 IPTASK va 3 

3 

50 Information Type I ¢ “4 

51 I] 4 

52 III 4 

53 IV 4 

| 

60 Read (Type I) Display 4 
61 II : 
62 II 3 
63 IV i 


80 Decide to Change Display 
11 Visually Access Keyboard 
20 Manually Access Keyboard 
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30 Key In (Param. Set Choice) 
10 Visually Access Display 
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60 Read (Type I) Display 
II 
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63 IV 


81 Verify Change has been impiemented 
correctly 


12. Visually Access Keyboard 
21 Manually Access Keyboard 
31 Key INSERT (P.S.) 
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Figure 3-26 Keyboard TAM Effect/No Effect 
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3.3.7.1 Validation of Model Behavior (Cont'd) 


The final perturbation to the standard mission data was the allow- 
ance of excess time for the completion of the mission. The results of these 
data changes are as follows. 

% a 

Keyboard TAM (EFFECT SIM) had tasks 1, 50, 52, 53, 54, 60, ‘él, 63 
marked dormant. Keyboard TAM (NO EFFECT) had tasks 1, 50, 52, 53, 60,. 61, 63 
and 12 set dormant initially. Using SAINT's parameter modif ication capabili- 
ties, the completion of task 80 was specified to cause tasks 10, 40, 51 -and 
62 to be made dormant in the NO EFFECT TAM (these tasks are used to check the 
display before it is inserted). The rest of the data were identical for both 
TAMS. The results of the TAMs are provided in Table 3-1. 


Table 3-1 


NUMBER OF SUCCESSFUL 
TIME OF COMPLETION (IN SECONDS) MISSIONS 


Mean = Mtitsow” «Max. «= Std. 
Effects Sim: 13.4 0.0. 67.0 6.42 95/100 
No Effect: 8.1 0.0 14.9 2.53 106/100 


The data supplied by the keyboard (EFFECT SIM) TAM were inserted in- 
to RTP leg 4 (i.e., the keyboard tasks in the RTP were assigned parameter 
sets with values of 13.4, 0.0, 67.0 and 6.42 seconds). These parameter sets 
were assigned to tasks 1 to 6, 87, 92, 112 to 114. The results of 100 itera- 
tions with these data are found in Table 3-2. 


Table 3-2 


NUMBER OF SUCCESSFUL 
is a COMPLETION (IN SECONDS) MISSIONS 


Mean: “Sin. Max. Std. 
3401.4 2999.4 3811.3 105.60 77/100 


The data from the keyboard (NO EFFECT) TAM were then substituted for 
the keyboard (EFFECT SIM) TAM data. Thus, the parameter sets describing 
tasks 1 to 6, 87, 92, 112 to 114 now have values (in seconds) of 8.1/mean, 
0.0/minimum, 14. 9/max imum and a standard deviation of 2.53. The remainder of 
the data is identical to the standard mission. The results of 100 iterations 
of these data are provided in Table 3-3. 


3.3.7.1 Validation of Model Behavior (Cont'd) 
Table 3-3 


NUMBER OF SUCCESSFUL 
TIME OF COMPLETION (IN SECONDS) MISSIONS 


Mean Min. Max. Std. 
3274.0 2997.6 3374.8 152.31 100/100 


The next perturbation in the standard mission data was an increase 
in the operator speed factor. The value of the speed factor was decreased 
from 1.0 to 0.8. Thus the time taken for these operators to perform a task 
is decreased by a factor of 0.2. The remainder of the data is identical to 
the standard mission. The results of 100 iterations of these data are found 
in Table 3-4, 


Table 3-4 


NUMBER OF SUCCESSFUL 
TIME OF COMPLETION (IN SECONDS) MISSIONS 


Mean Min. Max. Std. 
2695.4 2658.7 2729.0 14.78. 100/100 


The values for tne operator stress thresholds were changed next. 
The previously used value of 2.5 was Changed to 1.9, The remainder of the 
data is identical to the standard mission. The results of 100 iterations of 
these data are in Table 3-5. 


Table 3-5 


; NUMBER OF SUCCESSFUL 
TIME OF COMPLETION (IN SECONDS) MISSIONS 


Mean Min. Max. Std. 
4567.3 4124.8 4749.6 146.91 0/100 


The final data perturbation in the validation study was change in 
the time allotted to each operator for the completion of the mission. The 
time allotted (3400.00 seconds) was increased to 4000.00 seconds. The rest 
of the data remained identical to the standard mission. The results are in 
Table 3-6, 
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NUMBER OF SUCCESSFUL 
TIME OF COMPLETION (IN SECONDS) MISSIONS : 


Mean Min. Max. Std. 
3166.6 3133.7 3215.4 14.41 100/100 


Discussion and interpretation of these results is presented in para- 
graph 3.3.7.1.1. 


3.3.7.1.1 Analysis of Results. The Task Summary Report and the Mission Sum- 
mary Report have provided data required to analyze the results of perturba- 
tions of data (paragraph 3.3.7.1). As can be seen from Table 3-1, the key- 
board TAM, which was used when there would be no effect on the simulation, 
consumed an average of 5.3 seconds less than the keyboard which would affect 
the simulation. This was an expected result because parameter changes caused 
four tasks in the NO EFFECT TAM to becume dormant (i.e., use zero time). 
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The task which logically followed (in view of the static and dyna- 
mic evaluations) was the insertion of these TAMs into a leg of the RTP. Leg 4. 
was arbitrarily chosen. When the values for the EFFECT SIM keyboard TAM 
(Table 3-1) were used for each keyboard task in leg 4 of the RTP, the values 
in Table 3-2 resulted. A comparison of the values of Table 3-2 and Table 3-3 
(NO EFFECT) showed that the NO EFFECT TAM used 127.4 fewer seconds from the 
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total mission time than did the EFFECT SIM keyboard TAM. Table 3-7 shows | 
that the NO EFFECT TAM caused 8.3% less stress on operator 4. These results 
were further enforced by the fact that the RTP containing the NO EFFECT TAM 
was 100% successful, while the RTP with the EFFECTS SIM TAM was only 77% suc- } 
cessful. In this respect, the model proved itself to be extremely sensitive gS 
to the type of data perturbations which the evaluation of implementing dif- ‘ 
ferent interfacing techniques would cause, thus establishing itself as an 4 
effective design evaluating tool. ; 
Table 3-7 i 
AVERAGE OPERA- MEAN MISSION SUCCESSFUL | j 
TOR STRESS TIME MISSIONS s 
(OP #4) PERIOD A 
RTP=EFFECT SIM KEYBOARD TAM 1.61 3401.4 77 4 
(STANDARD MISSION) j 
RTP-NO EFFECT KEYBOARD TAM 1.47 3274.0 100 | 
INCREASE OPERATOR SPEED 1.07 2695 .4 100 
DECREASE STRESS THRESHOLD Zeal 4567.3 0 


EXCESS TIME 1.12 3166 .6 100 ie 


The change (by 0.2) of the operator speed decreased the mean 
stress value on operator 4 by 0.54. The mission time decreased by 706.0 sec- 
onds. Also, the maximum stress moved. from task 48 to task 57, 


Thé lowering of -the stress threshold caused an increase (0.76) in 
the stress on operator 4, Due to the disorganizing effect cf the high 
stresses, this mission exceeded the standard mission by 1165.9 seconds. The 
maximum stress tasks.moved from task 57 to 66. 


The allowance of more time for the mission caused a decrease of 
0.48 in the stress value of operator 4. Because the operators were not fre- 
quently under a stress greater than the assigned stress threshold, the stress 
had an organizing effect shown in the fact that the time for completion of 
the excess time mission is 234.8 seconds lower than that of the standard RTP. 


3.3.7.1.2 Conclusions. These results validate the developed model as a us-~- 
able tool for the evaluation of an I0S. The model is sensitive to perturba- 
tions of input data. This sensitivity will greatly aid a designer in evalu- 
ating an IOS concept. For evaluation purposes, the RTP with the EFFECT SIM 
keyboard TAM is the most beneficial. The changing of a single parameter set 
in only 11 tasks of the RTP changed the average stress on the operators, the 
number of successful missions, and the average times of completion. These 
results are invaluable in evaluating or choosing interface methods, 


The results of the run with excess time show the effects of in- 
creasing the time allowed for a mission. This will tremendously aid an 
evaluator or designer, who can now choose the best interface method -to fit 
not only stress characteristics, but also time limitations. 


The runs with an increased operator speed factor and decreased 
stress thresholds show the effect operator characteristics have on the mis- 
sion, The operator characteristics can now be taken into account in the de- 
Sign or evaluation of an JOS. 


3.3.7.2 Validation of Model/Real World Conversions. The key to an evalua- 
tion lies in the TAMs. Although the TAMs have been treated as building 
blocks for the RTP, they are also independent entities capable, in them- 
selves, of evaluating I0S interface devices. 


The parameter sets needed to quantify the tasks comprising the TAMs 
were chosen from PARAMINDX.DAT (Appendix G). A time to perform the particu- 
lar I0S activity was estimated, and this amount of time was allotted to -the 
TAM. The TAMS were then run and their results were compared with an actual 
trial performed on a jet trainer IOS. 


The TAM chosen to validate the "model to real world conversions" was 
keyboard TAM 18 (adjacent CRT). 


The time data used to quantify the task within the TAM were obtained 
either experimentally or from ergonomic literature. 


The parameters assigned to the tasks within the keyboard were as 
provided in Table 3-8. 
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3.3.7.2 Validation of Model/Real World Conversions (Cont'd) 


Table 3-8 


q ty OFS 


Mean i Max 
Visually Access Display 1.78 
IPTASK - 
Information Type II’ 1: 
Read Info Type Ti? 1; 
Decide to Change Display 
Visually Access Keyboard 
Manually Access Keyboard 
Key In® 
Verify change has been 
implemented correctly 
Visually Access Keyboard 
Manually Access Keyboard 
Display Delay 


Key Insert” 
*NOTE: These parameters have been estimated. 


PIERCE, J., & Kailin, J. Reading Rates and the Information Rates of a 
Human Channel. Bell System Technical Journal, March 1957. 
Ibid. 


NEAL, A. Time Intervals between Keystrokes, Records and Fields in Data 


Entry with Skilled Operators. Human Factors, 1977, 12(2), 163-170. 
Ibid. 


Keyboard TAM 18 was used to call up a CRT page. The results of run- 
ning this TAM were (in seconds) 15.6/mean, 0.0/minimum, 78.3/maximum and 
6.83/standard. 


The minimum time of 0.9 occurred when the page to be called was al- 
ready displayed. on the CRT (no change was necessary). The maximum time was 
due to errors in key strokes which then necessitated the repetition of the 
TAM, 


The times -arrived at for experimental performance of the IOS activ- 
ity were (in seconds) 12.5/mean, ae 20.0/maximum, and 2.1/standard. 


These results were eralhae "using a subject who was naive to the I0S 
study and having him call up ‘a. CRT page. The minimum value determined exper- 
imental ly was higher than that Obtained from running the TAM. The explana- 
tion is that, experimentally, the page was aiways called up. The possibil- 
ity that the page had already been displayed was not considered. The maximum 


values also differ because thére was no stress in the experimental conditions. 


The discrepancy in the mean time between real world and model re- 
sults can be explained by the parameter sets. The parameters quantifying the 
tasks in the TAMs were actuai!, "best estimates." However, the model did 
prove its results to ke realistic with the input data. Thus, as the input 
data describing the tasks become more refined (or realistic), the output more 
closely approximates real life conditions. 


3.3.8 Summary. An evaluative process was developed which will accept from 
the user an easily generated quantitative description of a candidate inter- 
face and produce an objective assessment of the described interface. The 
evaluative model is objective and task-specific, developing suitability meas- 
ures of an interface to an instructor activity. 


Qualitative descriptors were defined quantitatively for insertions 
into the model. This was accomplished through four stages of development: 


(a) Instructional Task List (ITL). The ITL is a compilation of al] 


known activities an instructor may perform to set uf the simulator, 
present and participate in a training mission, and observe the 
trainee's performance. 


Representative Task Profile (RTP). The RTP is a typical training 


mission assembled by sequencing selected activities as presented in 
the I7TL. 


Instructor Task Description (ITD). The ITD is a further breakdown 


of the items of the RTP into the tasks which comprise them. 


10S Activities List. This list is the final link in the descrip- 
tive chain. It links the instructiye tasks of the RTP and the 
man-machine allocations of the ITD to the available/vesirable i0S 
devices. 
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3.3.8 Summary (Cont'd) 

The ITL lists the tasks an instructor may perform on an JOS. The RTP 
is an arrangement of some of the ITL tasks in lesson plan fashion. The ITD 
relates the RTP tasks to an I0S. The IOS Activity List gives interface meth- 
ods (TAMs) by which the ITD items «nay be performed. 

An evaluation is twofold: static and dynamic. The static evaluation 
covers the basic suitability of an IOS device to a given task. To coniplete 
the evaluation, the individual task-device combinations must be tested under 
realistic conditions in the context of the 10S interface as a whole. 

The pele device for a static evaluation is the TAM. TAMs are written 
to describe each action an ‘pete deter performs to accomplish a given task 
with a designated interface method. TAMs are structured to contain most ac- 
tions needed to perform the given task in several ways. As such, not all 
tunity to designate the necessary tasks active and the others dormant, thus 
always maintaining the structural integrity of the TAM. The tasks which are 
linked to build a TAM are: 

(e) Visuai access 

(f) ‘Manual access 

(g) Operation 

(h) Read 

(i) Decision 

(j) Information type | 

(k) Display qualifier 

(1) Attent-ion-getter. 
The human instructor is divided into seven operators: 

(m) Visual 

(n) Right manual 

(o) Left manual 

(p) Cognitive 

(q) Audio 

(r) Verbal 


(s) Pedal. 
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tasks comprising a TAM are always necessary. SAINT affords a user the oppor- 
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3.3.8 Summary (Cont'd) 


The dynamic phase of the evaluation is performed using the RTP. The 
TAMS are arrangéd- in a logical order to simulate a lesson plan. The basic 
construction of the RTP is similar to that of the TAMs, with the capability 
of active and dormant designations. The RTP is made up of modules, each of 
which are model. descriptions of one task described in the ITD. These modules 
are divided into three classifications: 


(t) Simulation control modules 
(u) Assessment modules 
{¥) Documentation modules. 


The tasks are mathematicaily described by parameter sets. The parame- 
ter set contains the mean time, maximum time, standard deviation, and.minimum 
time for the completion of a task. 


These values are fitted to one of 11 specified time distribution 
curves and acted upon by the operator's stress threshold, accuracy factor, 
and speed factor. A time is then assigned to each task. Parameter sets 
which are assigned te @ task need not remain with that task throughout the 
network. Thus, if the same task is repeated, but the times taken to perform 
it should be different with each repetition, the network need not be inter- 
rupted or corrupted. SAINT provides the capability of having the completion 
of one task trigger a parameter set change. 


SAINT's output is divided into benchmark and detail iterations. The 
benchmark iterations provide statistics on start and end times of the tasks, 
but do not take any of the operator characteristics (i.e., speed, stress, ac- 
curacy) into account. 
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The detailed iterations..provide outputs on start and end times for 
tasks, operator stress, task stress, cohesiveness, and operator idle and busy 
times, plus a summary of all 100 iterations. 
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The system was validated by running the keyboard TAM with two different 
data sets. The summary statistics from keyboard TAM I, which included the in- 
structor checking the display before pressing INSERT, were used as the para- 
meter sets for any task in the RTP (leg 4) requiring a keyboard to call up a 
CRT page. The same Jeg of the RTP was then run with keyboard TAM II, in 
which the tasks pertaining to checking the display were designated dormant. 
The summary statistics from TAM II were then inserted into the RTP in ptace 
of these frum TAM 1. As expected, less time was Consumed when TAM II was 
used. The stresses on the operators in the RTP were also smalier as a result 
of the substitution of TAM II. Operator speed factor was increased and the 
model behaved as predicted. The time taken for the mission and the average 
stresses on the operators was lower. When the operator stress threshold was 
lowered, the model also behaved predictably, showing greater task stress and 
operator stress, greater disorganization and a requirement for a larger time 
allotment. 
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4. APPLICATION OF OBJECTIYE METHODS 


4.1 Evaluation of Candidate Methods 


4.1,1 Introduction.to Detailed Research.Plan. The purposes of the detailed 
research plan are first to apply the developed evaluation methodology to 
three interface techniques and second to compare two different I0S designs 


intended for the same simulator. 


The TAM's and their analytical networks define the static physical as~- 
pects of the respective devices and provide input ports for ali pertinent fac- 
tors, e.g., display quality, information exchange rate and human performance 
parameters. The data obtained from these TAMS are inserted into the RTP which 
is then run as a benchmark task. 


| 
| 

The values for the parameter sets implemented in the TAMs are obtained I 

either through literature searches of ergonomic data or through best approxi- a 
mations when pertinent }iterature is not available (paragraph 3.3.7.2). 


The capabilities of the model are demonstrated using an RTP as a typi- 
cal I0S task, and impiementing it to evaluate first, two I0S configurations, 
and Subsequently, three candidate interfacing methods. 


The expected results of these tests included establishment of the use- 
fulness of the model, generation of comparative data for the candidate meth- 
ods, and provision of further information useful in developing new design con- 
cepts and IOS applications. 


. Be ee 


i 
4.1.1.1 Purpose and Scope of Plan. The purpose of the first part of the L. 
plan, comparison of two existing instructor operator stations, was to show 
the approach used to implement the evaluation methodology. The comparison 
shows -how to analyze [0S configurations using data supplied by the TAMs for 
the RTP, and how to analyze ana compere the results obtained. 


2 


ha aeons bn Sha. Seemed 


The purpose of the second part of the rlan, evaluation of inter- | 
facing techniques, was to demonstrate the suitability of three interface 
methods to the task performed on a particular IOS. Three interface methods 
were evaluated first, on their ow’ merit (in TAMs) and second, as an integ- 
ral part of an IOS on which a typical mission was performed. 
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The instructor/operator stations chosen for evaluation were two off- 
board configurations for a typical jet Fighter trainer (Figures 4-1 and 4-2). 
The rationale for this selection was to show how the different configurations 
of an [0S controlling the same simulator and performing the same mission af- 
fect the operator's performance. 


4.1.1.2 Expected Results. The research plan has been proposed to accomplish 


| 

i 

| 

| 

= 
the following: . : 


{a) To establish the validity of the modelling philosophy, structure, 
and mechanization of the RTP by actual -test and usage. 
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Figure 4-1 
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Instructor Station #2 


Figure 4-2 
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(b) To verify the implementation method whereby the user can transform 
qualitative real world data into modelling parameters in a step-by- 
step fashion, and demonstrate the utility and ease of access to the 
model output data. 


(c) To yield insight into the parameter sensitivity pattern in the man- 
I0S system by virtue of comparative performance readouts for the 
three candidate interface modules and two IOS configurations. 


(d) Te generate data upon which further development work may be based 
and which will support the discussion of new design concepts, as 
well as refinements of the modelling technique. 


4.1.2 Comparison of Two 10S Configurations 


4.1.2.1 Basic Approach. The two configurations are referred to as #1 and #2 
(Figures 4-1 and 4-2). The two configurations differ only in the placement 
of the CRT's; the hardware on #1 is identical to that on #2. The CRT's on #1 
are located at either end of the instructor console, separated by the panel. 
The CRT's on #2 are adjacent to each other on the left and center of the 
console while the panel is on the right third of the station. In each case, 
both CRT's are controlled by an auxiliary keyboard or dedicated push button. 
The arrows on the figures (4-1 and 4-2) indicate the instructor's position at 
the IOS. 


The evaluation was ‘performed using the RTP (Appendix B) as a lesson 
plan. The ITD (Appendix C), which is an expansion of the RTP, and the IOS 
Activities List (Append.x C}, both in the general form, were tailored to fit 
the instructor stations in question. This tailoring was accomplished by 
designating IOS activities as either active or dormant. This step was neces- 
sary because the 10S Activities List, associated with each item of the J70, 
was designed to allow for all possible methods in which that task could be 
performed. That is, not every I0S activity presented is applicable to the 
design of every instructor station. 


TAMs describing the activities pertinent to the console were then 
chosen. The tasks comprising the chosen TAMs were assigned parameter sets 
(Appendix G) and run in SAINT. In readying these TAMs for running, several 
assumptions were made for the sake of consistency. 


(a) Attention-getting tasks in the TAMs were always designated dormant 


(b) The instructor would always check the display to ascertain that the 
desired change had been implemented. 


The design of the jet trainer console confronted the evaluators with 
several constraints. These are: 


(c) Stylized instruments (CRT PG 600) could be called up only to the 
right video display unit (RVDU) 


: 
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(d) Scaling and centering of maps was a combination of keyboard and 
push button tasks. 


Due to the fact that the two jet trainer consoles differed only in 
the placement of the CRT's, the IOS activities were identical for buth #1 ana 
#2 with only the visual access times, manual access times, and the cumulative 
effects of these times differing between the two configurations. 


4.1.2.2 Test and. Demonstration Results. The TAMs (and their reference code 


es ne, 


letters} in which the differences between #1 and #2 surface are: 


(a) Kead_YAM 


(i) CRT adjacent to ins*ructor position (A) 
(2) CRT remote from instructor position (B) 
(b) Keyboard TAM 


(1) Keyboard accessing CRT cdiacent to instructor position (be- 
tween 5 and 12 keystrokes) (H) 


(2) Keyboard accessing CRi .emove from instructor position (be- 
tween 5 and 12 keystroxes} (&) 


(3) Single key ne. ucessing a display, e.g., HARD COPY (M) ’ 

(c) Push Button TAM q 
(1) Push button in bank accessing remote CRT (C) : 
(2) Unique push button accessing remote CRT (E) | ; 
(3) Push button in bank accessing adjacent CRT (D) | 4 
(4) Unique push button accessing adjacent CRT (F) | 

(4) Map TAM | 
(1) Push button panel adjacent to CRT, keyboard adjacent to CRT (1) | | 
(2) Push button panel adjacent to CRT, keyboard remote from CRT (J) | : 
(3) Push button panel remote from CRT, keyboard adjacent to CRT (K) | “4 
(4) Push button panel remote from CRT, keyboard remote from SRT (L) | 


The time parameters for the remaining tasks in the modules are . 
iden‘ ical for #1 and #2. The results of running the TAMs are shown in Ap- 4 
pendix G. These code letters will be used to identify these parameter sets 
throughout this section. 
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4.1.2.2 Test and Demonstration Results (Cont'd) 


The kayboard TAM (see Figure 3-26), both adjacent to and remote from 
display, assumed that the instructor checked the display to verify the change 
had been implemented. Tasks numbered 10, 40, 51, 62, 80, 11, 20, 30, 81, 12, 
21, 70 and 31 were activated for both types of keyboard TAMs. The parameter 
sets differed for tasks numbered 10, 11, 12 (all visual access. task). The 
values for locating and focusing on a CRT from an adjacent keyboard or on a 
oaeiee from an adjacent CRT were experimentally determined to be (in sec- 
onds ): 


Mean Max. in. Std. 


1.78 1.50 0.23 


The same values for the remote CRT were: 


2.40 2.00 3.00 


The number of keystrokes was greater than or equal to five in both 
cases and the time allotted for completion of the TAMs was 17 seconds. The 


results of running these TAMs with the above values are shown in Figures 4-3 
to 4-5. 


Table 4-1 restates the parameters obtained from running these TAMs. 
These parameters will be used in the RTP. 


Table 4-1 Restated TAM Parameters 
Time Task Completed 
(in seconds) Missions 


Mean Min. Max. Std. Successful Unsuccessful 


Adjacent 15.7 0.0 78.4 751 97 3 
Remote 16.3 0.0 40.9 5.24 75 25 


As can be seen from the table, a keyboard located adjacent to the 
CRT being addressed requires 0.6 second less access time than a remote key- 
board. Given the same amount of time to complete TAM, the IOS with the key- 
board adjacent to the CRT had a 97% successful completion rate compared to | 
75% for the remote configuration. A successful mission is one in which all 
tasks are completed in the prescribed maximum time. Unsuccessful missions 
result from insufficient time allotments which raise stress levels, thereby 
degrading the operator performance. ; 
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Figure 4-3 Keyboard TAM Adjacent - M 
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4.1.2.2 Test and Denionstration Results (Cont'd) 


The read CRT TAM also has different parameters associated with it, 
depending upon whether the instructor is reading the adjacent CRT or a remote 
one. Task 10, VISUALLY ACCESS, is the only task in-which the parameter set 
differs from the adjacent to the remote display. The parameter sets used to 
quantify the visual access time from the instructor position to the adjacent 
and remote displays are the same as those given for manually accessing the 
keyboard from adjacent and remote CRTs. A time of 35.0 seconds was assigned 
to the TAM. The results of running these TAMs with the above values- are shown 
in Figures 4-6 to 4-9, Table 4-2 restates the values obtained from the Mis- 
sion Summary Reports. These are the values which will be assigned to the READ 
tasks in the Instructor Task Description. 


Table 4-2 Restated Mission Summary Report Values (No. 1) 


Time Mission Completed 


(in seconds) Missions 
Mean Min. Max. Std. Successful Unsuccessful 
Adjacent 31.0 27.0 32.6 1.06 100 0. 
Remote 31.3 27.0 32.8 1242 100 0 


As can be seen from these results, the time required to read a dis~ 
play on a remote CRT is 0.3 second greater than that required for the adja- 
cent CRT. The number of successful missions was equal (100%) for both dis- 
plays. 


The manual access task in the push button TAM can have four different 
parameter sets associated with it. The selection of the proper parameter set 
was dependent on whether the push button was isolated or in a bank, and remote 
from or adjacent to the CRT which displays the push button controls. The 
parameter sets are shown in Appendix G. The manual access task also has four 
parameter sets associated with it. These tasks are dependent -on the same con- 
ditions as the visual access tasks. The parameter sets for tasks 10 and 20 
differec as a function of push button and CRT control. There was always a 
verification of the new displays. The time allowed for this TAM to be comple- 
ted was 15 seconds. The results of running these TAMs with the preceding data 
can be found in Figures 4-10 to 4-17. Table 4-3 restates the values obtained 
from the Mission Summary Report. 
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Figure 4-6 Read TAM Adjacent - Task Summary 
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4.1.2.2 Test and Demonstration Results (Cont'd) 
Table 4-3 Restated Mission Summary Report Values (No. 2) 
Time Mission Completed 
{in seconds) Missions 
Mean Min. Max. Std. Successful Unsuccessful 
Bank-Remote 17.5 0.0 93.7 14.51 67 33 


Adjacent 12.0 0.0 38.9 6.19 88 12 


Isolated-Remote 11.8 0.0 96.6 10.89 93 
Adjacent 10.3 0.0 36.0 3.76 99 


An isolated push button (or a bank of no more than three push but- 
tons) which controls the CRT displays adjacent to the push button panel re- 
quires the minimum time and has a 99% success rate, 


These newly determined values were used as parameter sets each time 


the I0S activity they described appeared in the preconstructed model of the 
RTP. 


Prior to running the model of the RTP, time requirements for each 


leg had. to be set. An experienced instructor pilot estimated the time re- 
quirements for each leg of the RTP and, further, the time necessary for the 
completion of each module in an RTP leg. Those tasks within the modules 
which were to be carried on throughout the duration of the module (i.e., cog- 
nitive, monitor intercom, make written notes) were assigned times equal to 
the estimated duration of the module. The time requirements for the other 
tasks were supplied by the TAMs. The IOS activities which differed from 
configuration #1 to #2 were identified. These were reading activities and 
accessing activities. 


The identification was performed by stepping through the RTP in 
order to determine on which CRT a display was called. This was facilitated 
by the fact that stylized instruments can appear only on the right video dis- 
play unit. Thus, if a second display was needed at the same time as the 
stylized oe display, it nad to be called on the left video display 
unit (LVOU). 


Several standards were set to preserve consistency throughout the 
evaluation. The standards set for configuration #1 were: 


3 = a Ste ee ee 
5 vee fits teekt at tt Re nae eae PO a eae 


4.1.2.2 Test -.and Demonstration Results (Cont'd) 


(e) 
(f) 


(g) 


The LVDU was considered remote from the keyboard and the instructor 
position (Figure 4-1). 


The RVDU was considered adjacent to the keyboard and the instructor 
position. 


Both CRTs were considered adjacent to the one push button panel. 


The modelling standards set for configuration #2 were: 


Both CRTs were considered adjacent to the keyboard and the instruc- 
tor position (Figure 4-2). 


The LVDU was considered remote from the push button panel and the 
instructor position. 


The RVDU was considered adjacent to the panel and the instructor 
position. 


In RTP 3, the tasks in which these differences are apparent, the 


parameter sets assigned are: 


#1 Parameter Set Task #2 Parameter Set 
G 1 H 
G 2 H 
A 43 B 
A 37 B 
A 50 B 
A 27 B 
A 121 B 
A 43 B 
G 6 H 
G 7 H 
D 68 C 
A 69 B 


A description of these tasks can be found in Appendix B by cross- 


referencing the task number with the ITD. A complete printout of the results 
of RTP leg 3, configurations #1 and #2, can be found in Appendix I. 


(k) 
(1) 


a ea tr 


The following parameters remained constant for #1 and #2 in RTP 3: 
Time allowed for mission : 1860 seconds 
Operator speed factor : 1.00 
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4.1.2.2 Test and Demonstration Results Cont'd 
(m) Operator accuracy factor: 1.00 


(n) Operator stress threshold : 2.50 


The following tasks were initially declared dormant for #1 and #2: 


77 4) 103 101 
Re 78 49 105 102 
- 80 52 110 
a 81 56 117 
83 26 67 
84 29 70 
86 33 71 
ey: 88 89 76 K 
Bot 42 92 95 ‘ 
a 48 93 97 4 
z 34 94 100 8 
% The results for #1 and #2 are shown in Figure 4-18 (see Appendix J “ 
x for the full printout). The Mission Summary Report shows 65 successful mis- _ 3 
BG sions for #1 as opposed to 78 for #2. The task summary report shows times of & 
ae completion of leg 8 of the RTP to be: 4 
Ba ad 
% Time Task Completed (in seconds) i; 
F Mean = Min, «= Max, «= Std, 2 
ut fl 1846.7 1764.8 1927.2 30.20 4 
q 
: #2 1834.1 1762.6 1895.0 28.91 } 
: : 


By changing the parameter sets of the tasks specified above from 
those of #1 to those of #2, an average of 12.6 seconds was saved from the 
average time necessary to complete leg 3 of the RTP. These 12.6 seconds made 
the difference of 13 more successful missions for #2. Stress levels were 
also lowered for the operators on #2 as a result of the new configuration. 
Table 4-4 shows a comparison of the stress levels of each operator on #1 and 
#2 (see paragraph 3.2.3.12 for operator identification). 
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Figure 4-18 #2 Leg 3 (4 of 6) 
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Figure 4-18 #2 Leg 3 (5 of 6) 
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4.1.2.2 Test and Demonstration Results (Cont'd 


- Table 4-4 Comparative Stress Levels (Leg 3) 


#1 Mean Stress Operator Number #2 Mean Stress AStress 
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As can be seen from the AStress column, the greatest decreases in 

stress affected operators 1, 3, and 5 (visual, right manual, audio). The re- e 

sults cari be explained by. analyzing the difference between #1 and #2. The i 

difference lay in the placement of the two-CRTs and the panel. Thus, the two 4 

adjacent CRI configuration (#2). was better suited to the lesson: presented | 

here. 4 

‘a 

‘ The visual operator on #2 had less head and eye movement to perform, 3 

and the cumulative effect of this lowered the stress level on that operator. 4 

& 

Qperator 3 (right manual) also had less stress in the #2 conf igura- A 

tion. The amount of time operator 3 was. idle, while operator 1 jooked to a 4 

remote display (#1), decreased on #2. AS a result, less time was taken from wg 

i the total time allowed operator 3, keeping the stress level down. Operators “G 
; 2, 4, 5, 6, 7 (left manual, cognitive, audio, verbal; pedal) also experienced ai 
P lower stress. Because none of the tasks involving these operators differed if 

from #1 and #2, thé lowering of the stress here is explained as follows: the j 

decreased stress on operators 1 and 3 lowered the disorganizing influence A 

these two operators nad on the other factors. As a result, the stress levels “4 


on the other operators decreased. Conclusions .drawn from these results wil] 
follow the presentation of data from the other legs of the RTP. 
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In RTP 4, the tasks in which the differences between #1 and #2 -are 
apparent, the parameter sets assigned to them are: 
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4.1.2.2 Test and Demonstration Results (Cont'd) 
#1 Parameter Set Task #2 Parameter Set 
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The following parameters remained constant for #1 and #2 in RTP 4: 
Time allowed for mission : 1600 seconds 

Operator speed factor : 1.0 

Operator accuracy factor : 1,0 

Operator stress threshold : ara 

The following tasks were initially declared dormant for #1 and #2, 


either because they were not needed, or because they did not apply to the jet 
trainer configurations: 


111 60 88 
115 66 89 
116 77 90 
42 78 92 
48 79 93 
49 80 94 
50 81 95 
52 83 97 
54 84 100 
56 85 101 
57 86 102 
127 


ae te de renee ye ea pe UR tg NE ENE te RE OU OTe cen aT Bo Sm PRL SEES Oi AP Ce II, LYING EY PRA OOTY SN 
Sears ain eae ene . 7 


La e 
ao vad 
seg )hcprrahina scinhtese.. Cf pt aesteiean etnias tommaiebitiB dha taka tsd a erectlaanea hanes sttambosn en Pacenihtcc tite Stace a AialsietA Sect e eea Sg t t T Abe ee sh eh Sa 


4.1.2.2 Test and Demonstration Results (Cont'd) 


SAINT's results for running #1 and #2 are shown in Figure 4-19. 
The Mission Summary ‘Reports show 86 successful missions for #1 as 
opposed to 92 successes for #2. The task summary report shows the times of 
completions for leg 4 of the RTP to be: 
Time Mission Completed (in seconds) 
Mean Min, Max, Std. 
#1 1415.7 1261.3 2032.2 185.44 


#2 1373.9 1249.6 2030.7 157.78 


By using the parameter sets for #2 instead of #1 in the tasks speci- 
fied above, the average time necessary to complete leg 4 of the RTP was re- 
duced by 41.8 seconds. These 41.8 seconds were responsible for the six extra 
successes on #2 and also for the reduction of the stress levels encountered 
by: the operators. 


The new configuration (#2) was instrumental in lowering the stress 
on the operators. Table 4-5 shows a comparison of the stress levels of each 
operator on #1 and #2. 

Table 4-5 Comparative Stress Levels (Leg 4) 


#1 Mean Stress Operator Number #2 Mean Stress AStress 


1.17 1 1.11 0.06 
1.13 2 1.09 0.04 
Lie BC. 3 1.09 0.04 
1.13 4 1.09 0.04 
1.47 5 1.40 0.07 
1.41 6 1. 36 0.05 
1.41 7 1.36 0.05 


- The stress appreciably decreased on all operators. The changing con- 
figuration had a direct impact on the stress of operators 1, 2, 3, 4 (visual, 
right manual, left manual, cognitive). The greatest disparity between the 

two consoles surfaced in tasks 112, 113, 114 which on #1 entailed the manipu- 
lation of a page remote from the controlling keyboard. Not only were these 
tasks time consuming, but the Task Summary Report shows the number of fail- 
ures of these tasks on #1 to be nearly seven times greater than the same tasks 
performed on #2 under identical conditions. 
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Thes2 results can be explained. 
tors had less movements to perform. 


The tasks in which the differences between #1 and #2 become clear in 
leg 5 of the RTP are: 


#1 Parameter Set 


J 


>rponr YP rPrrremr yr FG FS 


The following parameters remained constant for #1 and #2 in leg 5 of 


Time allowed for mission : 


Qperator speed factor 


4.1.2.2 Test and Demonstration Results (Cont'd) 


Both the visual and manual opera- 
The cumulative effect of these movements 
detracted less from the total allowed time causing less stress, fewer inis- 
takes and, therefore, more successful missions. 
enced a reduction in stress as a result of the lowering of the disorganizing 
influence of operators 1, 2 and 3. 


Task #2 Parameter Set 


1.0 


Operator accuracy factor: 1.0 


Operator stress threshold : 2.5 
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4.1.2.2 Test and Demonstration Results (Cont'd) 


The following ‘tasks were initially declared dormant for #1 and #2, 
either because they were not needed or because they did not apply to the jet 
trainer configurations: 


Tasks 


95 
98 
99 


2) 

25 

57 89 
60 92 
63 93 
66 94 
123 67 


SAINT's results for running #1 and #2 are shown in Figure 4-20. 
The Mission Summary Reports show 70 successful missions for #1 and 
80 successes for #2. The Task Summary Report shows the times of completion 
for leg 5 of the RTP to be: 
Time Task Completed (in seconds) 
Mean Min. Max. Std. 
1470.9 1263.3 1778.9 121.19 


1457.8 1294.3 1799.2 107.71 


By uSing the parameter sets for #2 instead of #1, the average :time 
necessary to complete leg 5 of the RTP was reduced by 13.1 seconds. In #2, 
these 13.1 seconds resulted in 10 more successful missions than #1. 


A comparison of the average stress felt by the operators in #1 and 
#2 is shown in Table 4-6. 
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4.1.2.2 Test and Demonstration Results (Cont'd 


Table 4-6 Comparative Stress Levels (12g 5} 


#1 Mean Stress Operator Number #2 Mean Stress AStress 


1.54 1.48 0.06 
1.29 Lee 0.04 
1,29 1.25 0.04 
1.36 Lisl 0.05 
1.03 1.02 6.01 
1,00 0.99 0.Q1 
1.00 0.99 9.01 


The stress decreased on all operators, but especially on operators 1 
to 4 (visual, right and left manual, cognitive). The configuration of #2 eli- 
minated keyboards controlling remote CRTs (tasks 100, 125) and all ov the re- 
mote readings. The cumulative effect of these tasks mcre than countered tasks 
68 and 5, which in #1 were push buttons on an adjacent panei, but i. #2 were 
now remote from the instructor position. Ouce again, operators 1 to 4 real- 
ized a lower stress on themselves which, in turn, lower:d the stress on the 
other operators. 


The tasks which exposed the differences between #1 and #2 in leg 6 
of the RTP are: 


#1 Parameter Set Task #2 Parameter Set 


86 
62 
2 


97 


cop) 


> Prmpmrepa rT nTrRr DD YP YFP aryYr so 
owwrweaoereVmenmiTtrt Tt WoO AKA DB 


4.1.2.2 Test and Demonstration Results (Cont'd) 
#1 Parameter Set Task #2 Parameter Set 


G il H 
118 H 
12 H 
50 B 
13 K 
60 B 


The following parameters were kept constant for both #1 and #2 in 
leg 6 of the RTP: 


Time allowed for mission ; 2500.0 seconds 
Operator speed factor ‘ 13:0 

Operator accuracy factor : 1.0 

Operator stress threshold : 250 

The tasks declared dormant in leg 6 of the RTP are: 


77 102 
78 34 
88 41 
57 
59 
60 
66 
95 


The SAINT results for the running of #1 and #2 are shown in Figure 


4-21. 


The Mission Summary Reports show 80 successful missions for #1, as 
opposed to 83 successful for #2. The Task Summary Report shows the times of 
completion of leo 6 of the RTP to be: 
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Figure 4-21 #1 Leg 6 (1 of 6) 
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Figure 4-21 #1 Leg 6 (2 of 6) 
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~ “TASK” SUMMARY REPORT ~~ 
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Figure 4-21 #2 Leg 6 (4 of 6) 


148 


wet eZ 


& 


Hate Me ie 


Ma ate SE ty OE Satin, A Tae 


"5 
a3 
Ag 


Roe 


2 Reber # he as CI en SRE RE 


5 A ee 


28 


Rok. Al sv 


~ 


erent 


thas 


& 


“2: 


2 


rier ye Ltda ad 


* 


2 
ter SR AS 


SE 


cries 


SOT RET eT SR ee we 


Ne of ae A a SS a eee eee 
RRORK ARO 0000 wale nOORKKROOHRNARNOORARAR 
monanimene socccapon enero on nmnee nnn 
TOTES COOSQIIO COCO OS STOOLS AST TO OLE SES, 
SO0000000000000 00000 FORO Ran MOONY 
SSSSSSSESSSOSSSSGoSSONAUNSSOAGCOORCNNRNR 
SeddddddoddddddKdddddEddddddodddOdddddd 
QQaLTs BOQQQOOOIASOOONANTIQagmaMNII AME RO 
OO Qn Hn DDCOOCOSOSOGOSSONANMGOIRE T#RGOOKR 
nididedpdddodsiddodotiiedesodn-pidrann 
BONTTRNESSOSSSCOSIOONANNOON GANT OOOONIOT 
Soddddddoddddodo III nina SdSSOSS SHON 0 
sgasgegesssssesassseseseesasenesasesses 
MKeeTNMNHMN re OS COCO SN 

rn) 


Seabed L Sebedenbend Ll CQO aa emnnmQOOOCONNNANSNNVOAND 
ecocoocoococoesse SSSooonsoooNOGOOOOGO~VVGS 


Wee’ 
Hee ide nlite net cterinicuditiniticinine ein ceieiieceiel 


VHON0 OF 190000 ~0000 nH TC OMMNNNAOE Aah OEGO-~ 
MOONAAUNGOOSOOCOOOGOONINOT AMT ONGOING 


SOCAN TNNDOCOCONMANAMWOaRKsEMNOSHAETOCAROS 
HON GITNVSOOOSOCOOOONNKAANNONNISONTAOTOT 


146 


movoownnnoaneeon ANTE TO COO OE IO CONS Ser 
NOSSO HHA OOND Gavidichbviniccawamevin denne acids 


346. ice Cues 


Fe) 
NOvnood VY {oanvoorseqHonHoNGrcanrogee tT 
i Ivlvlvlvivly! a mmm QOOODOOL So TOOTTOVITNONHNN 
Cucucucucucutucy ' x i USNC YTY me me UCI me mE CEN 


Sate ie Te iw ae eB at ete t] 
re-ddouNidcwONCicib ~duoddex oGwmdivivinigARRAN 
NIDIF DOH VANE TONRT WN 00S mENrONNCT IOMIG 
Pole kntotalelote) 906 OCINANNOMULS St TS 
{ : F Ret tet rit ! 
useenmasinlleods Hesonosemokan seoNaenhes 
HOTT IBHOON HF INDIORNGAGOAAIONDA=0F OOO 
NAOH SRAGGIOSOGNGOO TOR AGOGO mH ORGAoO 
DOnGDOO soso Ge2uNooNeTbAUNe 4eOOCS 
PE COIS MAROON G0090000NR ARN 
OMTNNMVTOOZORMRO nonnesconnadnaananeare 
andar denDosid candi deine ads an ¢gorndonice 
QAFAVIRA ATH AGNDOMOHOIID NEV AREA IGG ING 
e000c000| 1 AMOOOSCODOOC ERIM stu Cy 
eee | f 7 { Dine ' Arte 
i ; 
DONOONUUMOCORN DION ANE TEROO OAT O OTE ae tare 
QNTOOTO SQG=ANHOHO COOMPOMUNRAD-DI450 
ankoon 0 NICOe Se SeeMNDOIAIOae CTOTeT 
mA aaa NADDIGVIISCSOPSCONTINON 
HMRC} ' cecucy VA wet on cn cameo SCTCICUON 
ARARARRCOOONN =~ afr Ve COMCODGDNARAMMGS 0000 
Ponconnososaonnnrignveccedonod dnnNOGURAN 
Row aeOe" AAC TAME ED NID aneeehnor¢sgJODD 
ae oe VSOGONNNNNNAMserrey 
0:0:00-0r0 
AUN HTNMNODSCOV-DVOMN-CONODOVOETOOON-VHHES 
NAVOUDIDOCOND HaKRMOOKRAHOODHVHOBDOVDHs YNEMSO 
ANTANT THCODMVODAOCOOKA DDO OO PRA MON AONE 
OOROONA— PmARRO DIS FOOONNT INGDOO 
DQOQN VOD i OMNOANIMNNYDCODIOVOR VARA 
ROANKADDCOOORDE ONE VERKAGOOONNNNNAGNAVE Oro 
mAN HRA MOOD HAGAN TNE DONNNM«<ChNAT ERAN NONGD 
B8250a89} NANTON ORD NARTDOD aa NNNEOMDOG VOW 


=“COSSCOHSDVADDDDAM wt ar yn 
eto mt ca { MCINACEN f mietesovetes 
‘ 


j } ; | 
| 
flolalolalelololelolololelololejlolalotololelolelololelolololale] alelolololelo] 


: t 


oocooecobtioncoorbooocooopedooooRsoRon Co 


4 ‘ 


3} 

a } i ~~ 
; is 3 : : 
SCODCOOGONMOKNOOOMONONO2ROOHONDSOOCOHOKRONOSS 
SSOOSSHSCOOHHSSOSHSSSSOSSUSONOOOASOONOLSO 
POA DI $e we wt ot eee mee emt UNNI T TTS STITOMOINST 
[ololofololololololololololalolololellovelololololalelolelolololalelolalolare] 
696665006550S6556505660050005000560600000 
. intel tntet et esniet ed pied nietel niet Metetrteteteeeted pirértet rasa eentanet ated 
BVOQVQOKMORFRR EERE ERR EERE RE EEEREOUREEEE EEE 
22222 48Z 2455S ISI ss dd SZ de ss ddd 
ee OP PEEREEELEERELEETELULEEELEEEEE 
NUNUNEDELELELELE PEELED EERE COLE ER EE eee 

| { ‘ t 
POANOTM GRECO KUM TNHIDROM“AMEHORDSOKNRDESHY 
DYDD OGOOKAARE DOOODODODE SHPO SS OHSOCO mmm Keer 


Figure 4-21 #2 Leg 6 (5 


149 


{ 

q 

| 
Aa tasar 4 i 


Br 
2 Hae ectd Ss SRE 


rat: 


fbivsed ots EEE 


: 


ne 


peditacard tn treres SEED 


ry 
Stee eit 


g. 


: 


sane ae 
Pokievcs 


ee 
SE 


' 
f 
i 


of 6) 


ee: 


¢ 


+ 


a ace ects 
Pia sake 20am 


FAR 


a x 
eer 


t 
: 


| 
bi 
1 
i 
a ~~ 
ito) 
qe 
i Q 
{ wo 
—— 
} 
| wo 
} mm 
: w 
; a 
oO 
~w ws 
Ey ed 
Sd A Pav a ee : ““SNOISSIW WNsSSa99nSNA ZT - 7 . = 
- . -~- SNOISSIW “INsSSRIINS EB a p 
00'2 IZ"l 40°0 08 | 6 ‘BSBZ 2 B22 2 SG2t vy 'Leve & StS2 s'OBbI F'SZIL G‘OoZIZ E bE + FOE Fb b & 9IE co cose Z 
OO‘e IZ‘ 20‘O OB ‘I §'8SBZ S'S622 S S2t b LEtA S.6bGS S‘OBST b'Gel G‘Ozte E‘bte b'90E b& b 6 ‘STE GO 00S2 9 @ 
t?°2 26°2 200 ab’2 4 'ASG2 2 B22 2 S2I b LEbe tT 'bO!t 1’BEo Bbet TEL2° 3B 269 Z OFF1 1 |S E ¢951 00 ocse Ss S 
wow 29 G SL S. 9E 0.80.52. ELEA.2/2IEE ¢ LEL 6.26962. £°845. $ 69¥.01ZE .Z2.11S = 2 Bike 1 SbL41 2 “bt 2.9ret 00 Q0S2 “ws ke. > 
GS"2 "pF "M12 O° 96"l” Gv Eiée 9 elce & CEL “E Lobe 6 BFL O 419°" & be “GS EB9 OC OCEZ2 3 1961 Z bet 3B 24921 OG 00S2 c nH 
B62 oF 1 120 96'l BES 9 SIES » LEI E ZSbe 0894 O 419 &6 be ¢ 689 O OES t 1961 2 Sbt B L921 00 CdSe 2 a= 
249°2 94't 61'O BOS BEES 9'ZIES b LEI € ZSbe £669 6 146 9°22 S'€E9 ¢ 1922 2'991 S$ Bet ¢ Ezar 00 O0Ge 1 ie 
‘XYW ‘NIW G1S NV3W “XV ‘Ni agr.s Nv3tl “XU "NIW "ais NvSWi ¥VW Itt “GLS NY3H BWVIUvVAY YSEWNN q 4 
waH---SSBULS@m ann — Hen Rmn-GESN BWIL Wi0l------  ----+-+-- BwiL BuGT --------. beseee =--SWIL ASNE----- --e- BYIL yolvusdO \ 
fa ave cies thes ate : — .. .LUOdBY ABYHHOS NGISSIW 8. boast, chee ue Bhs? Sedlatecde Bok 
? 
| 
4 
* 
BS 
J 
— ery . ef ae eegener~empieninge: Sper A vawashs See eed Does , 
Se ast an CROSS aac eee ——— wipers ee 


SE Se ee MEYERS Ba 
FEE SIE TOE PE Te SE DI Re Pe Tage RS oe ee 
ae eee TS Koza SE Oe is 2am ea ae are PN aS aoe, an TANS a 
. - of * 1 
i ni ie 


~ aa as Paes * 


= na odessa Se ae = SN RES Se eS mated a ‘ ose s de gta) eteatet: See mnt 
4.1.2.2 Test and Demonstration Results (Cont'd) 
Time Task Completed (in seconds) 
Min, Max. Std. 
#1 2320.6 3064.6 153.74 
#2 2312.6 2913.8 137.44 
The average time to complete leg 6 of the RTP was 38.7 seconds great~ 
er for #1 than for #2. This time saving in #2 is due mainly to the fact that 
the keyboard and the instructor position at the console are adjacent to the 
two CRTs. There is no remote display. Thus, inserting changes to a page via 
keyboard, under a time limit, is done more efficiently when the keyboard and 
the CRT are adjacent. 
A comparison of the average stresses felt by the operators on #1 and 
#2 is shown in Table 4-7. 
Table 4-7 Comparative Stress Levels (Leg 6) 
#1 Mean Stress Operator Number #2 Mean Stress AStress 

2.19 : a 2,08 0.11 

2:03 2 1.96 0.12 

2.08 3 1.96 0.12 

2419 A 2.08 0.11 

2 43 5 2.42 0.01 

1,79 6 4.80 0.01 

1.79 7 1.80 0.61 

ae Once again, there #s a great decrease in the stress associated with 
, operators 1 to 4. fhe logic for this decrease in stress is the same as has 


previously been discussed, \ 


In RTP leg 7, the parameter sets differed from #1 to #2 in only 
three tasks. ‘hose tasks are: 


#1 Parameter Set Task #2 Parameter Set 


G 1 H 
A 43 B 
A 50 3 
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4.1.2.2 Test and Demonstration Results (Cont'd) 


The operator stress threshold, accuracy factor and speed factor re- 
mained the same as in the other legs of the RTP. The time assigned to leg 7 
was 600 seconds. Only these tasks were designated dormant: 


42 70 
48 76 
49 89 
52 2 
56 94 
67 


The results from running data of #1 and #2 are shown in Figure 4-22. 


The Mission Summary Reports show 100 successful completions for both 


#1 and #2. The Task Summary Report shows the times of completion of the leg 
to be: 


Time Task Completed (in seconds) 


Mean Min. Max, Std. 
#1 524.0 500.0 544.0 9.36 
#2 534.6 505.5 576.0 15.57 


The difference between #1 and #2 is that the operators of #1 average 
10 seconds less to complete the mission than those on #2. The stress levels 


for each operator are shown in fable 4-3. a 
a 8B 
Table 4-8 Comparative Stress Levels (Leg 7) B 
is 
#1 Mean Stress Operator Number #2 Mean Stress AStress ; 
‘g 
1.06 1 1.05 0.01 4 
1,62 2 1.61 0.01 | 
1.02 3 1.01 0,91 of 
1.02 4 1.01 0.01 “ 
0.99 5 0,99 0.00 
9.97 6 0.97 0.00 
G.97 7 0.97 0.90 
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Figure 4-22 #1 Leg 7 (1 of 4) 
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Figure 4-22 #1 leg 7 (2 of 4) | 
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Even though the operators of #1 took 10 seconds less to complete RTP 
7, the operators of #2 had lower stress levels. RTP 7 is performed when the 
aircraft is under TOTAL FREEZE, i.e., all parameters are frozen. Neither the 
instructor pilot nor the trainee feel any pressure. This was the reason for 
the large amount of time that was assigned to RTP 7. The leg of the RTP is 
a at i short to show the cumulative effects of the different features 
on and #2, 


The results of leg 3 to 6 of the RTP show a sum of 32 more success- 
ful missions using the #2 configuration. 


The times taken to complete a leg of the RTP were consistently lower 
for the #2 configuration than for #1. 


Operators on #2 were subject to lower stress levels than the opera- 
tors on #1. 


Configuration #2 is clearly the better design for an instructor sta- 
tion. This agrees with the opinion of the user who asked to have the origina} 
#1 layout modified to the #2 configuration at their cost. 


4.1.3 Candidate Interface Methods. Three I0S devices were used in the demon- 
stration tests: 


(a) Keyboard/CRT action 
(b) Dedicated push button/CRT action 
(c) Light pen/CRT action. 


Each I0S device was used in the implementation of an ITD item. The ITD 
items used were ACTIVATE DISCRETE MALFUNCTION and CALL UP MAPPING DISPLAY. 
The discrete malfunction item entailed the calling up of the malfunctions page 
on a CRT and activating the desired line. This task was performed by each of 
the candidate methods. The mapping display item vons.sted of setting a scale 
for tna map and then centering the map. tach candidate method was used to 
perform this task. 


The fact that the keyboard and tne direct action push button are two 
of the most often used 10S devices provided the rationale behind their selec- 
tion as candidate interface techniques, Both may be employed in simple and 
complex input tasks. The keyhoard has great flexibility and the push button 
represents quick system response and functional complexity limited only by 
the software capability available at the [0S. 


The light pen is a device suitable for both simple and complex tasks “¢ 
in connection with a CRT display. Its greatest advantage is in the capability iis 
of makiny precise and limited changes in a complex pattern, such as mapping i 
or tactical display, where the task time and the risk of input error must be 
minimized, 
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4.1.3.1 Basic Approach. The existing analytical networks, i.e., TAMs and 
R7P, represent typical activities and conventional IOS devices and functions 
related to a representative task sequence. 


The existing TAMs describing the use of a keyboard, push button, and 
light pen were fitted with the appropriate parameter sets (see paragraph 
4.1.3.2) for the jet trainer console. The tasks, within the TAMs which call 
for the instructor to verify that the new data input is correct, were always 
activated. The TAMs, which constitute the static phase of an investigation, 
established the times necessary to perform a task using a particular inter- 
face method, thus establishing the basis, i.e., time requirements, operator 
stress levels, by which the suitability of the device could be determined. 


The data generated by the TAMs, i.e., mean, maximum, minimum times 
of completion of the TAM, were incorporated into the model of the RTP to form 
the dynamic phase of the evaluation. The purpose of the dynamic phase of the 
evaluation was to determine the suitability of the interface device as an 
integral part of an entire IOS console. In order to perform this evaluation, 
each leg of the RTP was examined to determine the number of times discrete 
malfunctions had to be called up, and then the number of times mapping dis- 
piays were used. Leg 6 of the RTP calls for the activation of a discrete 
malfunction six times, and for the calling up of maps twice. Thus, RTP leg 6 
was chosen as the vehicle for the dynamic testing. 


In both the static and dynamic test phases, the operators were 
asSiqned values of 1.0 for their speed and accuracy factors, and 2.5 for 
their stress threshold factors. These are the values Siegel and Wolf con- 
sider to be average for an operator.’ 


The crucial point of the evaluation was actually the dynamic phase. 
One time was assigned to the RTP leg 6. it was then run three times, each 
run containing the parameter set values for a different interface device used 
to activate discrete malfunctions. 


Tne effect the diffarent interface methods had on the total perform- 
ance of the mission surfaced in the stress levels of the operators, the aver- 
age time necessary for completion of the leg of tne mission and the number of 
successful missions per run. This procedure was then repeated implemen’ ing 
the candidate interface methods in the mapping display tasks of RTP leg 6. 


4.1.3.2 Suitability fest and Results. The candidate methods which were sub- 


jected to evaluation were: 
(a) Kevboard/CRT action 
(b) Push button/CRT action 


(c) taght pen/CRT action. 


Siegel, A., Wolf, J., et al. Modification of the Siegel-Wolf Operator 
Simutation Model for On-Line Experimentation. Aerospace Medical Researcn 


Laboratory, Wright-Patterson Air Force Base, Ohio, June 1971. 
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These methods were evaluated with respect to their suitability to 
perform two common instructor tasks, activating discrete matfunctions and 


arranging mapping displays. For a description of parameter sets, see Appendix 
G, PARAMINDX. 


4.1.3.2.1 Interfaces/Mao Displays. The keyboard TAM was the first to be 
evaluated (Figure 4-23). Task 1, ATTENTION~GETTER, was not used on the 
assumption that this was an instructor-paced task. INFORMATION TYPE III 

(task 52) and READ TYPE III (task 62) were relevant to the evaluation so they 
were activated while the other types were made dormant. The CRT was adjacent 
to the keyboard, therefore tasks 10, 11 and 12, VISUALLY ACCESS, were assigned 
parameter values for distance ecual to 12 inches (Appendix &), i.e., a mean 
access time of 1.78 seconds. MANUAL ACCESS tasks (20 and 21} were assigned a 
mean time of 1.00 second (distance equalling 12 inches). 


The IPTASK was assigned an arbitrary value of 0.1 second, while 
DISPLAY DELAY (task 70) had a value 1.0 second. INFORMATION TYPE IIi and 
READ TYPE III were assigned mean times of 4.0 seconds and 3.0 seconds respec~ 
tively. Task 30, KEY IN, was assigned a mean time of 1.35 seconds, This task 
calis up the proper page on the CRT. Task 31, KEY IN, scales and centers the 
map. This takes between 5 and 12 keystrokes. The time assigned to this task 
was 3.1 seconds. A total time of 26 seconds was assigned to the mission. 


The results of this test were 97 successful missions out of 100 
(see Figure 4-24). The greatest mean stress felt was on operator 4 (cogni- 
tive). This stress was 1.37, which is well below the operator stress thres- 


hold (2.50). The time (in seconds) for completion was 23.1/mean, 3.1/minimum, 
36.5/maximum and 5,17/standard. 


The keyboard was a suitable interface device for this task. 


The push button TAM (Figure 4-25), in order to perform a mapping 
display task, requires three buttons to be pressed. One button is dedicated 
to calling up the map to the CRT, a second to scaling the map, and the third 
to centering the map at the aircraft. A designer would have these buttons 
grouped together. Such a configuration allows the evaluator to treat these 
buttons as if each one were isolated. Therefore, the mean time assigned to 
task 30, DEP&tSS PUSH BUTTON, was three times the mean time necessary for one 
button. This tripled value is 2.7 seconds. The ATTENTION-GETTER (task 1) 
was not necessary because this TAM was run at the instructor pace, INFORMA- 
TION TYPE II and READ TYPE II were activated (tasks 51 and 61). These types 
represent the information displayed on a push button. Through SAINTs para- 
meter manipulation ability, the completion of task 30 triggered a parameter 
set change. Tasks 51 and 61 were set dormant, while tasks 52 and 62, INFORMA~ 
TION and REAQ TYPE III, were activated. These new information types represent 
the information display2d on the CRT (maps). VISUAL ACCESS, MANUAL ACCESS, 
DISPLAY DELAY and IPTASK parameters were kept the same as they were for the 
keyboard TAM, 


The results of this test were 99/100 successful missions (Figure 
4-26). Operator 4 (cognitive) felt the greatest mean stress. Heowever, this 
stress value (1.02) was wel] below the operator stress threshold (2.50). The 
time for completion of this TAM averaged 15.9 seconds. 
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KEYBOARD TAM 18 
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Figure 4-23 Keyboard TAM 
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Figure 4~24 Keyboard Report 
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4,1.3.2.1 Interfaces/Mep Displays (Gant'd) 


The minimum time of 0.0 seconds was due to the possibility that 
the desired display was already on the CRT and no changes were needed. 


The use of the light pen (Figure 4-27) to arrange a mapping dis- 
play consisted of calling up a map from a menu page, choosing a scate (the 
scale factors were assumed to te on the map itself) and moving a cursor in 
order to center the map on the aircraft. Thes2 actions were incorporated 
into tasks 36, 31, and 32. As in the two previous TAMs, INFORMATION TYPE III 
and READ TYPE III were active. The MANUAL ACCESS, VISUAL ACCESS, IPTASK, and 
DISPLAY DELAY times were the same as in the two previous TAMs. A time of 26 
seconds was allotted for this TAM. a 


The results of the running of the light pen TAM with the preceding 
data were 62 successful missions out of 100 (see Figure 4-28). The time (in 
seconds) for completion was 31.1/maan, 0/minimum, 130.9/maximum and 23.39/ 
standard. 


The mean stress on both the visual and cognitive operators (3.49 
and 3.64) exceeded their stress thresholds (2.50). 


Table 4-9 presents a summary of the values obtained for each candi- 
date interface method, 


Table 4-9 Vaiue Summary 


Keyboard Push Button Light Pen 


Mean time of completion (seconds) 23e1 15.9 shed 
Maximum Average stress 1.37 1,02 3.64* and 

3.49% oe 
Successful missions/100 97 99 62 a 
Time allotted for mission (seconds) 26.0 26.0 26.0 | 


* NOTE: Both these values exceed the operator stress threshold. The least 
suitable device was the light pen. While the other two interface 
methods were able to allow the tasks to be completed in under 26 
seconds, while never reaching the opeator stress thresholds, the 
light pen method exceeded both the allotted time and the operator 
stress threshold. 


While the keyboard showed itself to be an acceptable interface 
method for thes2 tasks, the push button method was preferable. It required 
7.2 seconds less to perform the mapping task by push button than by keyboard 
and, while both techniques did not stress the operators greatly, the push 
button method results in the lower stress value. 
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LIGHT PEN (SINGLE/MULTIPLE TARGET/MOBIFY COMPLEX DISPLAY) 1AM 16 
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This, however, was not the complete picture. The devices were id 

then tested dynamically as an integral part of the I0S console. These tests | 4 
are described in paragraph 4.1.3.3. i 
mee 

4.1.3.2.2 Interfaces/Activates Discrete Malfunctions. The keyboard TAM os 
(Figure 4-29), when used to perform the task cf activating a discrete mal- es 
function, contains the same parameter as it did for mapping displays, with 4 
the exception of tasks 36, 31 and information and read types. Task 30, KEY i ¥ 
IN, called for the instructor to display the malfunction page and to address fe 
a line. This was done using a series of five keystrokes. According to PARA- | 4 


MINDX (Appendix G), a mean time of 1.35 seconds was assigned to this task. 
Task 31, KEY INSERT, consisted of depressing an isolated push button on the 
keyboard and was assigned a time of 0.02 seconds. INFORMATION TYPE II and 


READ TYPE II (tasks 51 and 61) were activated. A time of 17 seconds was 
assigned for this mission. 
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The results of rurnaing this TAM with the preceding data were 97%. 
successful (see Figure 4-30}. The greatest mean stress felt was on operator 
4 (1.29) This value was 1.2] lower than the operator stress thresnold. The 


time for completion of the mission (in seconds) as 14.4/mean, 12.1/minimum, 
72.5/maximum and 7.67/standard. 


YY Veen 
ry 


ar? aa 


* ~ + * 
“ ri A. ts ont “ pm 
Cy Sees 


The push button TAM (Figure 4-31) was a simple one button opera-~ 
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tion. The malfunctions were printed on the cedicated push buttons which were a 
arranged in a bank, When the malfunction was activated, the button color 3 
changed. As a result of this, there was no need for the instructor to look : 
to the CRT to verify that the change was implemented. There was no parameter ie 
modification of the INFORMATION TYPE I! and READ TYPE II tasks, as when a push j 
button interface was used to call up a mapping display. The interface itself § 
was used to verify that the change had been made. Task 10, VISUALLY ACCESS & 
PUSH BUTTON, was assigned a time of 1.78 seconds. Task 20, MANUALLY ACCESS “a 
PUSH BUTTON, had a time of 2.1 seconds and described the time taken to posi- 3 
tion the instructor finger over the correct malfunctior button. The actual i 
depression of the button took 0.02 seconds. A total time of 15 seconds was Roe 
allowed for completion of the TAM. 1% 
Be 
This TAM yielded 99 successful missions (Figure 4-32). The maxi- B 
mum stress on an operator was 1.07 on operator 4. The time required for com- é 
pletion of this TAM (in seconds) was 10.4/mean, 0/minimum, 32/maximum and 3 
3,.44/standard. “4 
The light gen TAM (Figure 4-33) assumed that the malfunction page 4 
was already displayed on the CRT. The instructor needed only to touch the a 
light pen to the CRT area sensitive to the activation of the desired ie 
malfunction. This TAM yielded 95/100 successful missions (Figure 4-34). The 
average stress on an operator was highest on operator 4 (1.08). The time 


required for completion of this TAM was 11.4 seconds. 
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The minimum time of 0.0 seconds represents the possibility that 
the malfunction was already active. 
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Table 4-10 presents a summary of the values obtained for the three 3 

interface devices: 4 
4 

Table 4-10 Vaitue Summary é 

Keyboard Push Button Light Pen “ 

Mean time of completion (seconds) 14.4 10.4 11.4 ; 
Maximum Average stress 1.29 1.09 1.08 : 
Successful missions/100 97 99 95 Ff 
Time allotted for mission (seconds) 17.0 15.0 11.4 é 
ia 

The least suitable device was the keyboard, although its perform- 4 

ance was quite satisfactory. The stress on the operator was, at most, just 4 
under half of the operator stress thresheld. The light per was the most 3 
Suitable interface method. Its completion time was the most task-efficient 4 
of the three devices (see paragraph 3.2.1), and the stress on the operators é 
was lowest. 3 
a 

The light pen's showing was close to that of the push button with & 

respect to time, but had a 5% failure rate as opposed to the push button 1%. 4 
4 

The integration of these devices into the IOS console is described 3 

in paragraph 4,1.3.3.1. 3 
or) 

4.1.3.3 Dynamic Test and Results. The next phase of the evaluation was the | ; 
incorporation of the three candidate interface devices into the dynamic | J 
model, i.e., RTP. Leg 6 of the RTP was chosen because of the number of times sy 
the discrete malfunctions were activated and the mapping displays were called. a 
The time or completion statistics obtained through the running of the two ITD 1 
items using the three interfacing techniques were used as parameter sets for Lae 
those tasks in the RTP. The RTP leg 6 was assigned a time of 2500 seconds for i | 
every run. The RTP and the parameter sets were the same ones used for the | 3 
evaluation of the #1 configuration (paragraph 4.1.2). | . 
5 

4,1.3.3.1 Oynamic Test/Mapping Display. Tasks 1 and 13 in leg 6 cf the RIP 
required the calling up, scaling and centering of a map. The parameter set a 
for the keyboard TAM was inserted into the RTP for tasks 1 and 3. The values é 
of this parameter set, in seconds, 23.1/mean, 3.1/minimum, 36.5/maximum, and | \ 
5.17/standard. | 3 
The branches from tasks 1 and 13 indicating successful completion : 

of the tasks were assigned a value of 1.97. The branches taken if the task ae 
failed were assigned vaiues of 0.03. a 
|g 

The results of running the RTP with the keyboard interface handling 4 

the map display tasks are shown in Figure 4-35. There were 77/100 successful 3 
missions, Task 1 failed four times and had to be repeated. Task 13 failed 4 
once and was repeated. The mean stress values on the operators were: j 
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4.1.3.3.1 Dynamic Test/Mapping Display (Cont'd & 
+4 

Operator 1 - 2.22 ? 

Operator 2 - 2.11 4 

Operator 3 - 2.11 4g 

Operator 4 - 2.22 3 

Operator 5 - 2.44 ; 

_ Operator 6 - 1.79 i 

’ Operator 7 - 1.79 r 

The times of completion, in seconds, were 2507.9/mean, 2363.4/mini- 4 

mum, 3071.3/maximum, and 155.57/standard. | 34 
The parameter sets for tasks 1 and 13 were then changed to those ! i 


parameters obtained from performing mapping display tasks via push buttons. 4 
The probability of successful completions of these tasks was set at 0.99 and aes 
that of failure at 0.01. The results of this RTP (Figure 4-36) were 80/100 


successful missions. Task 1 failed only once, while task 13 has no failures. 
The mean stresses on the operators were: 


Onerator 


1 - 2.19 ee 
Operator 2 - 2.08 2 
Operator 3 - 2.08 $ 
Operator 4 - 2.18 : 
Operator 5 - 2.43 3 
Operator 6 - 1.73 3 
Operator 7 - 1.78 % 
The times of completion, in seconds, were 2493.8/mean, 2318.4/mini- : 
mum, 3061.8/maximum, and 154.39/standard. Fi 
The parameter sets of tasks 1 and 13 were again changed, this time 4 
to those parameters obtained from the light pen TAM. The probability of suc- _ 3 
cessfully completing tasks 1 and 13 was set at 0.95 and that of failure at 7" 
0.05. The rest of the RTP remained unchanged. i 
The results of the pen parameters in the RTP were 71/100 successful : 
missions (Figure 4-37). Task 1 failed three times and task 13 failed once. 4 
The mean stress values on the operators were: ; 
Operator 1 ~- 2,25 é 
Operator 2 - 2.15 a 
Operator 3 - 2.15 3 
Operator 4 - 2.25 s 
Operator 5 - 2.48 f 
Operator 6 - 1.80 4 
Operator 7 - 1.80 


a3 


The time for mission completion, in seconds, was 2528.9/mean, 
2355.3/minimum, 3112.0/maximum, and 167.78/standard. 


A summary of these results is shown in Table 4-11. 
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Figure 4-37 Light Pen Report (1 of 3) 
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Figure 4-37 Light Pen Report (2 of 3) 
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Figure 4-37 Light Pen Report (3 of 3) 
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4.1.3.3.1 Obynamic Test/Mapping Display (Cont'd) 


ee ap a 
Pa ~ Table 4-11 Value Summary 
SL : . Keyboard Push. Button Light Pen 
7 a 
Mearl, + ime of compet ion (seconds) 2507.9 2493.8 2528.9 
Mean stress /ongeator 1 2.22 2.19 2.25 
2 2.11 2.08 2615 
3 2.11 2.08 ray bs) 
4 oeee 2.18 2.25 
5 2.44 2.43 2.48 
6 1.79 1.78 1.80 
7 1.79 1.78 1.80 
Successful missions/100 77 80 71 
Time allowed for mission (seconds) 2500 2500 2500 


In the comparison of the use of the three interfacing methods to 
arrange mapping displays, the push button technique yielded the best results, 
The average time of completion was 6.25 seconds less than the allowed time 
while both other methods exceeded the allowed mission time, The push button 
interface had the highest percentage (80%) of successful missions. The opera- 
tors felt an average of 2.36% jess stress than the operators using a light 
pen, and an average of 0.87% less stress than the operators using a keyboard 


The keyboard was better suiced than the light pen to the mapping 
task, but not by the large margin by which the push button surpassed the key- 
board. The keyboard average time of completion was 0.31% greater than the 
assigned time while the light pen exceeded this time by 1.15%. The keybnard 
had a 6% higher success rate than did the light pen. The keyboard. operators 
Felt 1.29% less stress Ehan the light pen operators. 
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In view of lhese results, of the three candidate interface methods, 
the push button interface was the best suited device for arranging a mapping 
display, followed by the keyboard, The light pen interfacing method was un- 
Suitable. 


4.1.3.3.2 Dynamic Test/Activate biscrete Malfunction. Discrete malfunctions 
are activated four tines” in leg 6 of The RTP.” Task 118 is the activation 

task. The parauecer sel from the keyboard/maifunction TAM was inserted into ; 
the RTP for task i118. The probability of successful completion of task 118 ' 
was set at 0.97 and the probability of failure at 0.03. 


ee en erent miata 


The results of runuing the &kTP with the keyboard interface hand- 
ling the activation of discrete malfunction tasks are shown in Figure 4-38 
There were 88/100 successful missions. Task 118 failed 98 times. However, 
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Figure 4-38 RTP Leg 6 Keyboard Report (1 of 3) 
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Figure 4-38 RTP Leg 6 Keyboard Report (3 of 3) 
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4.1.3.3.2 Dynamic Test/Activate Discrete Malfunction (Cont'd 


it should be remembered that this task was performed four times per mission. 
Thus, it failed 98 times out of 500 releases. The mean stress values on the 
operators were: 


_ Operator 1 
Operator 
Operator 
Operator 
Operator 
Operator 
Operator 


The time of completion, in seconds, was 2432.6/mean, 2290.8/mini- 
mum, 2830.1/maximum, and 100.3/standard. 


The parameter set for task 118 was then changed to the parameters 
obtained from the light pen/malfunction TAM, The probability of successfully 
completing this task was set at 0.95 and that of failure at 0.05. 


The results obtained from running the sixth leg of the RTP with 
these data (Fi gure 4- 39) were 97/100 successful missions. The average time 
of completion, in seconds, was 2400.2/mean, 2273. 7/min imum, 2713.0/max imum, 
and 2.28/standard. 


The average stress values on the operators were: 


2.04 
1.91 
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2.38 
1.77 
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Operator 
Operator 
Operator 
Operator 
Operator 
Operator 
Operator 
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Task 118 failed 100 times in the 500 times it was released. 


The parameter set for task 118 was then changed to the parameters 
obtained from the push button/malfunction TAM. A probability of 0.99 was 
assigned to the successful branch of task 118, with the failure branch having 
a probability of 0.01. 


The results obtained from running the RTP with these data (Figure 
4-40) were 96/100 successful missions. Task 118 failed 97 times out of 497 
releases. The time of completion, in seconds, was 2397.7/mean, 2304.4/mini- 
mum, 2692.2/maximum, and 63.3/standard. 
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if Figure 4-39 RTP Leg 6 Light Pen Report (1 of 3) 
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Figure 4-39 -RTP Leg 6 Light Pen Report (3 cf 3) 
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Figure 4-40 RTP Leg & Push Button Report (3 of 3) 
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4.1.3.3.2 Dynamic Test/Activate Discrete Malfunction (Cont'd) 


The operator stresses were: 


Operator 1 - 2.03 Operator 5 - 2.38 
Operator 2 - 1.91 Operator 6 - 1.77 
Operator 3 - 1.91 Operator 7 - 1.77 


Operator 4 - 2.03 
A summary of these results appears in Table 4-12. 
Table 4-12 Vaiue Summary 
KEYBOARD PUSH BUTTON LIGHT PEN 


Mean time of completion (seconds) 2432.6 2400.2 2397.7 
Mean stress/operator 1 2.05 2.04 2.03 

2 1.98 1.91 1.91 

3 1.98 1.91 1.91 

4 2.09 2.03 2.03 

5 2.40 2.38 2.38 

6 1.77 bets 1.77 

7 1.77 Lf 1.77 
Successful missions/100 88 97 96 
Time allowed for mission (seconds) 2500 2500 2500 


The keyboard, although not unsuitable for this task, was not as 
Suitabie as the push button and light pen. Its time of completion was 67.4 
seconds under the allowed mission time, and the operator stress values were 
all under the stress thresholds of 2.50. The stress values, however, were 
2.68% greater than those of the push button and light pen operators. 


y 
LBS Prinses 


The light pen and the push button interfaces were both suitable to 
the malfunction task. The time they take to perform the mission differs by 
3.6 seconds and the stress value for the operators is almost the same. 


In conclusion, in performing mapping tasks the push button inter- 
face was most effective, followed by the keyboard, The light.pen was unsuit- 
able. In performing the discrete malfunction task, al] three interfacing 
methods were suitable, but the keyboard was least effective while the light 
pen and push button interfaces were equally suitable. From these results, 
the choice of candidate methods should be, in decreasing effectiveness: 


«poles rnatnadenincrnoatitbir nes ass BoE 


(1) Push button 
(2) Keyboard 
(3) Light pen. 
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4.1.4 Summary. Two evaluations were performed. The first evaluation com- 

pared two different I0S configurations. The second evaluation was performed 
in order to compare the suitability of three candidate interface techniques 

to two commonly performed instructor tasks. 


The evaluation of the two I0S configurations was twofold: static and 
dynamic. The static phase of the evaluation consisted of choosing TAMs which 
described the IOS activities pertinent to the jet trainer console. The tasks 
comprising ‘these chosen TAMs were assigned parameters from Appendix G 
(PARAMINDX). The appropriate tasks were then designated dormant, time 
allotments were assigned to the TAMs and they were then run by SAINT. The 
results of these TAMs were then used as parameter sets for the dynamic phase 
of the evaluation. 


The dynamic phase consisted of running the validated RTP using the 
data obtained from the static phase. Each. leg of the RTP was run twice, once 


with the TAM data from configuration #1, and once with data from configuration 


#2. The data generated from these RTP runs were analyzed with respect to 
number of successful missions, stress values on the operators and time of 
completion. A summary of these results is given in Table 4-13. 


Table 4-13 Comparative Summary 


#1 #2 
Time for completion (seconds) RTP3 1846.7 1834.1 
RTP4 1415.7 1373.9 
RTP5 1470.9 1457.8 
RTP6 2496.0 2457.3 
RTP7 524.0 534.6 
Average operator stress (on RTP3 2.25 (OP#4) 2.23 (OP #4) 
operator registering higest) RTP4 1.47 (OP#5) 1.40 (OP#5) 
RTP5 1.54 (OP#1) 1.48 (OP#1) 
RTP6 2.19 (OP#1&4) 2.08 (OP#1&4) 
RTP7 1.06 (OP#1) 1.05 (OP#1) 
Number of successful. RTPS 65 78 
missions/100 RTP4 86 92 
RTP5 70 80 
RTP6 80 83 
100 


RT P7 100 

ae ae. 

The #2 configuration surpasses that of #1 in number of successful 
missions, lighter stress and less time used. 
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The evaluation of the candidate interface techniques was done in dyna- 
mic and static phases. The static phase consisted of using the three tech- 
niques (keyboard, push button and light pen) to perform two instructor tasks 
(averaging map displays and activating discrete malfunctions). The results 
showed that, for the map arrangement, the push button TAM was the most effec- 
tive, then the keyboard. The light pen proved to be unsuitable. For the 
activation of discrete malfunctions, all three techniques were suitable, 
However, the keyboard was the least suitable interface, while the push button 
was most effective, followed closely by the light pen technique. 


The parameters obtained from running these TAMs were then implemented 
into leg 6 of the RTP for dynamic testing. Each of the three interface 
methods were inserted whenever a mapping display was needed. The order of 
effectiveness of the interface methods, based on number of successful mis- 
sions, average stress on operators and time necessary for completion were (1) 
push button, (2) keyboard, (3) light pen. 


The parameters obtained from the malfunction TAM were then implemented 
in the RTP. The order of effectiveness of the interface methods, based on 
the same criteria as above, were (1) push button and light pen, (2) 
keyboard. 


In conclusion, if hardware were to be implemented on an IOS in order 
to perform the above two tasks, the light pen, although most effective for 
activating discrete malfunctions, is unsuitable for map arranging and is thus 
the least desirable. The push button interface is most suitable for both 
tasks, followed by the keyboard. 


4.2 Presentation of New Design Concepts. An on-board instructor/operator 
station (shown in Figure rar was analyzed and then was redesigned utilizing 
the concepts learned from the evaluation of interfacing techniques and off- 
hand I0S designs (paragraph 4.1). 


This 10S design consists of two CRls separated by a panel of push but- 
tons. A separate panel, containing the controls for the instructor inter- 
phone, is situated above one of the CRTs. The push button panel is divided 
into five subsections. The first section contains a real time clock and 
three potentiometers controlling the intensity of the panel lights, ceiling 
lights, and push button lights. The next panel contains a series of push 
buttons associated with slewing aircraft fuel tank quantities, and various 
miscellaneous functions. The third panel is the station/position panel. 

This panel provides the capabilities to control the status of ground radio 
stations. The simulator control panel contains controls for the motion sys- 
tem and emergency conditions. The last panel, the instructor audio panel, 
consists of toggle switches, push buttons, indicator lights and a volume con- 
trol which allows the instructor to communicate with the flight crew via pri- 
vate or regular aircraft communications. 
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Figure 4-41 On-Board [0S 
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4.2 Presentation uf New Design Concepts (Cont'd) 


In this I0S, as in most other commercial aircraft simulators, there are 
no repeater instruments because the instructor is in close enough proximity 
to the trainee to see the actual instruments. 


This design is simple, straightforward and functional. The keyboard al- 
lows easy access to the computer information which is displayed on the CRTs, 
while the push buttons facilitate the implementation of commonly used para- 
meters. 


However, aS a-result of the analyses performed (paragraph 4-1) on the 
jet trainer I0S, the information exists to improve this IOS. 


The jet trainer analysis showed that an IOS which requires jittle eye 
and head motion, and small travel distances to physically access devices, im~- 
proves the usability of an JOS. The trainer analysis of the three candidate 
interface methods revealed the light pen to be the least desirable while-a 
conservative arrangment of push buttons is the best with the keyboard as an 
all-purpose desirable device). 


The interface devices already implemented on this 10S are the keyboard, 
dedicated push buttons, and potentiometers. The push buttons are arranged in 
separate groups of 2 x 8 matrices. Although these are banks of push buttons, 
they are not so large that the instructor must search for a particular button. 
The labelling of the buttons is clear and easily read. 


These devices, however, can be employed further to enhance the usability 
of the 10S. The I0S designed with the newly obtained information is shown in 
Figure 4-42. The panel has been removed from between the CRTs, thus elimin- 
ating the tennis match effect of looking from the left CRT and across a panel 
to the right CRT (which was an undesirable feature of the jet trainer #1). 
The instructor can now visually access both displays with negligible eye and 
head movement. 


The new design also eliminates muca of the distance travelled in manual 
access tasks. The panel and keyboard are situated on a wedge in front of the 
CRTs (this is a very popular concept, used frequently in power generating 
station simulators). This arrangement (with the use of no-glare push but- 
tons) is visually more appealing, and manually easier to access due to the 
proximity of the interface devices to the instructor. The ‘desirability of 
this proximity was shown in the evaluation of #2 where, although visual 
access tasks were greatly facilitated, manual tasks become more difficuJt 
when the panel was placed on the far right-of the console. The removal of 
the panel from between the CRTs also eliminates the possibility of the in- 
structor inadvertently covering up -a vital display while trying to access a 
device. It must be strictly noted that if any interface device accesses only 
one specific CRT, that device will. be situated on the panel wedge directly in 
front of the CRT it accesses. For example, if approach plots and maps are 
most commonly called ud on the right CRT, the third panel (station/position) 
should be ia front of the right CRT. 
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Figure 4-42 -On-Board IOS (Improvec) 


4.2 Presentation of New Design Concepts (Cont'd: 


More extensive use of the interface devices chosen will] also improve the 
I0S from the point of view of the instructor pilot. While the keyboard is a 
very suitabie method of activating linés on a CRT page, direct line seiect 
buttons have proven (as in paragraph 4.1) to be more expedient. The relative 
simplicity of this design and the lack of large amounts of hardware allow 
this. Thus, the lines can be activated both by the keyboard and the push buts 
tons. Since there are 40 lines per CRT page, this bank of push buttons can 
be arranged as a 4 x 10 matrix. , 


A voice recorder, activated by a foot pedal or kneebar, is implemented 
in the new design to allow instructors to record notes (for their own use) 
while assessing trainee performance. This will not take the place of the for- —_ 
mal notes which must be made, but may replace the quick jottings instructors a 
make during an assessment. Each assessment module of the RTP contains .an on- 
going MAKE WRITTEN NOTES task which adds to the stress level of the emulated 
manual operator. The voice recorder will free the manual operator, thus low- 
ering the stress which will, in turn, add an organizing influence to the other 
emulated operators. This feature is also justified by the fact that, through- 
out the RTP, the least used emulated operators (also the operators with the 
lowest stress levels) are the verbal and pedal operators. 


These design concepts will have the effect of lowering the stress im- 
posed on the instructor pilot by making the tools of training more readily 
accessable, more easily usable, and more evenly spread over the instructor's 
basic numan capabilities. 
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5. CONCLUSIONS 


5.1 Model Effectiveness. The main thrust of this work was to develop an ana- 
lytical tool to assist in I0S- design and assessment. The too] was applied to 
two alternate IGS configurations and produced reasonable results. Clearly, 
the IOS model can be used with confidence, provided that sufficient data are 
available to describe accurately the componant devices of the 10S. 


In its present state, the model proves to be a useful tool in assessing 
alternate configurations or devices. With a larger data base, particularly 
ae in terms of a larger number of operators, it should be possible to use the 
2 model to establish design principles and constraints. For example, the model 
could be used to identify generically those functions suitable for dedicated 
push buttons and those suitable for keyboard/CRT. Based on model results, 
~ definitive statements on the use of color or layout of displays, etc., could 
Ee. |- be made. 


| 
| 
| 
Ps The limiting factor in Jef Thing the effectiveness of the model is exper- 
an iencé. The development of thé-modél is evolutionary since assumptions are 
4 made which can be validated only: by applying the model and reviewing the re- 
= sults. The fact that reasonable results were achieved in the two test runs 
py" serves to verify the assumptions made in developing the model. 
! 


Ph. 5.2 User Benefits. This section is to identify some of the benefits which a 
et. user may achieve through the Secor ‘the evaluation tool described in this re- 
Si port. It is anticipated that users-will include designers of instructor sup- 
: port systems similar to the I9S, evaluators or analysts of existing equipment 
: attempting to determine effect iveness or identify functional problems, and 
those upgrading or retrofitting existing equipment. 
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B Depending on the user's purpose, it may be necessary to examine or to 

af predict the performance of a given interface in a sequence of operative 
iy. tasks, to identify potential sources of time delay and generators of error, 
R or to determine the requirements for additional equipment or functional capa- 
E bilities to relieve the workload. 


The user may wish to analyze an existing design and determine its suit- 
ability and effectiveness in a new application, or to determine its capacity 
to accept additional tasks. It may be that the assignment of tasks between 
the instructor and the supporting subsystems and functions is in question, 


i1.8., the modes and methods cf operating the equipment may require examina- 
tion. 


} 

i 

i 

Paper-and-pencil studies usually fail to identify the many correlated 

factors and interactions, and their cumulative effects on operator perform- 

ye ance. The computer model yields both quantitative data and an overall quali- | 
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tative picture in which even subtle and subjective factors may be recognized. 
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5.2 User Benefits (Cont'd) eS 


In the earliest design phases, where functional requirements are ana- 
lyzed and man-machine task allocations are traded off against equipment and 
system limitations, the trends shown by the model outputs provide an excell- 
ent qualitative guide. Even the selection of the TAM's carries its own feed- 
back; their titles and definitions warn the user that a complex and high-work- 
load task is being described as the various command-display sequences are 
assembled in the model network, and that some form of machine assistance 
might be necessary. Running the model generates workload stress data on each 
of the hypothetical operators, such as the visual operator, indicating the 
nature of a possible problem in terms of unsuccessful tasks or missions. 


It is expected that as experience is guined, both the model and the de- 
signer's insight into its workings will develop to a point where an almost 
step-by-step design validation and optimization may be achieved. Even in its 
early form the model conclusively validated the request to change design spe- 
cifications on the jet trainer IOS by showing reductions in task time and 
work stress when the CRT displays were placed close to the opérating position. 
Furthermore, the results of that test indicated in a qualitative way that 
miniaturization of the IOS, the reduction in equipment size and distances 
separating controls and associated displays, is desirable in an off-board 
station from operational viewpoints as well as being desirable in terms of 
space utilization, particularly in the case of an on-board 10S. 


The user of existing equipment may want to analyze the workload presént- 
ed to the instructor by the I0S interfacing methodology in an attempt to im- 
prove the quality of training. The obvious error sources and ergonomic..pro- 
blems are easy to spot but latent workload factors are not. These may rob 
the instructors of the necessary time to perform teaching tasks. Instructors 
are usually reluctant to admit that the I0S is getting the best of them and 
will carry the unnecessary workload by exerting additional personal effort. 
However, the RTP (or a profile constructed specifically for the tasks in 
question) will specify the monitoring and assessment tasks required to truly 
achieve the training objectives and will indicate the high stress levels. 
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In order to make full use of the capabilities of a modern flight simula- 
tor, and the investment it represents, the instructor must be regarded as a 
major training asset as well as a potential limiting factor. If the IOS in- 
terface assists the instructor to take an active part and to exercise positive 
control over the training session, with sufficient time to monitor and coach 
the trainee, he/she will contribute significantly to the effectiveness and 
efficiency of skill development. If the opposite is true, the deficiencies 
will destroy the credibility of the simulation and cause wasted training 
time. Even in older devices, updating the I0S and instructor facilities can 
bring about improved training and trainee evaluation, or as the very minimum, 
increased utilization of the device and reduction in manpower requirements. 
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‘Using the model to identify existing shortcomings in an instructor 
facility will assist the user in deciding where to apply aailable funds. jl 
The model will compare alternatives and predict the expectad improvements in 

terms of instructor workload or additional training capabilities. A typical 
trend in this area is to eliminate a second instructor/operator by additional 

software and I0S equipment; the model will indicate the workloads carried by 
the eyes and hands of the first instructor and indicate the feasibility of 
one-person operation. 
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5.3 Recommended Further Work 
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5.3.1 General. The results presented in this report have been based on a 
very small sample of experimental data, an equally small ergonomic data base 

available in the literature, and on approximations derived from past. exper- 
jence where no other data could be found. Nevertheless, the model output 
seem to be reasonable and the trends logical and consistent with real world 
predictions. It is important, therefore, to validate the accuracy o* the 
model against real life conditions, and thereby calibrate this highly pro- 
mising analytical tool for future use. 


Further efforts should include the following principal areas: 


(a) Validation of model by test under real or realistic operating con- 
ditions 


(b) Refinements to the input/output interface, using the existing model 
ports, to include details of human perception and performance para- 
meters 


(c) Development of advanced 10S interface methodology and the corres- 
ponding TAMs to represent that methodology in the main modelling 
network. 
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5.3.2 Model Validation. The validation of the model should consist of quan- 
titative experimental runs, a general exploration of the visual world of the 
instructor, and specific tests to verify the results produced so far on such 
input equipment such as the light pen and the light gun. Validation. should 
include the comparison of time allowances used in the RTP to worst-case times 
in a simulated mission. The approximations used have been based on the ex- 
perience of a single, highly qualified, instructor pilot. While the basic 
trends are probably correct as shown, the data base is insufficient to pro- 
vide reliable results. 
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An eye point marker device, now available in-house, should be used to | 

investigate the visual world of the instructor, the modes and patterns in i 

which he/she uses displays and other sources of information. This device 

uses fiber optics and closed-circuit television to superimpose a light beam 

reflected from the eye pupil onto the background of the visual scene avail- 

able to the subject. The beam generates a mark which corresponds to the 

intersection of line of sight with the visual scene, i.e., the principal 

point of interest at any t?me. This method would be especially useful to 
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verify the conditions of visual access and eye-head movements necessitated by. 
the arrangement of the displays relative to the associated controls on the 


10S panels. Specifically, the comparison between #1 and #2 configurations 
would be easily validated by tnis method. 
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The eye point marker could generate new data on the relationships that 
exist between a solid bank of keys or illuminated push buttons, the same 
equipment arranged in functional groups or isolated by location, and: the CRT 
displays or remote indicator lights associated with them. The model. has the 
necessary input ports, but quantitative data are very scarce.. 


In addition to. being used as an analytical tool, the eye point marker 
could possibly. have application as an input device similar to a light pen. 


5.3.3 Model Refinement. The principal input ports to the model, i.e., the 
TAM networks, were implemented using macroscopic ergonomic or similar data. 
Since each TAM was first run as a stand-alone SAINT network, this powerful 
model presented few practical limitations to the depth of detail to which the ok 
components of each activity could be explored. Parameters of human perform- 8 
ance, cognitive processes and decision making. Factors were included in the 
network and. quantitative data generated to represent their influence. 


Further development of the model would jermit analys:is of parameters 
such as color coding, CRT page design, information exchance rate and noise. 
Combined with data from the eye point marker investigation, this analysis 
could pinpoint such differences as CRTs mounted in a vertical pattern against 
a horizontal arrangement where the relative distances are similar but the 
viewing angles are different. 


The typés and arrangement of ‘input devices couldé also be further ex- 
plored and described in the model. A large solid bank of push. buttons is in- 
stinctively disfavored, but color coding, local echo response and functional 


grouping with strong marking produce a workable keypack. The relative merits 
‘of these factors could be quantified by the model. 
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Another area of model development would be to expand and simplify the 
input/output activity. Designers must make many small inputs in their term- 
inology and, they desire quick and’ simple answers from the model. The basic 
framework for this ideal mode} response already existS but the facility of 


operation will come only with experiencé and the further definition of mode! ~ 
ing terms to the user. 


5.3.4 Methodology. The model has the capability to analyze new input/output 
channels and methods which are coming into use in I0S design. The voice in- 


terface and portable terminal devices are two worth further exploration and : 
optimization ‘by model Ting. 


The technology -of speech recognition and voice synthesis -has made tre- 
mendous strides in the past few years, both ‘in terms of quality and realism 
of communication. and in terms of informat-ion transfer capability and accuracy. of 
As part of this study, two existing systems. have been examined. One had a - 
vocabulary of 800 words and could handle inputs from four terminals at once. 
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The other, designed for computer-assisted training of air traffic controllers, 
can deliver prompting messages and recognize correct answers. The digital ‘sys- 
tem -then automatically assesses ‘the trainee operator's perfomance, points out 
weak areas and recommends remedial training. This facility would deliver an 
enormous command and teaching capability to the instructors, reiieve them of 
many communications tasks by delivering clearance messages, etc., and even 
evaluating answers, thereby enhancing the credibility of simulaticn. While 
the application of this technique would radically change the IOS philosophy, 
performance, and workload, the system would remain within the scope of the 
modelling and analysis techniques developed in this réport. 


Portable I0S designs and devices are relatively limited in control and 
display capability at present but further developments are underway. A typi- 
cal device carries two lines of 12 characters each and has a 20-key keypack 
with which it communicates to a large instructor support software and display 
system. The device is slow compared to a keyboard but combined with a lesson 
plan system where each step represents a complex 10S input, it is adequate to 
control a training session. Using light- emitting diode or plasma displays, 
miniature keyboards and a simplified presentation, an enhanced portable unit 
(perhaps. seat-mounted) could be developed. This would permit the instructor 
to assume the bent position for optimal viewing of the trainee while maintain- 
ing a reduced but adequate communication with the IQS. 


5.3.5 Conclusion. This study has initiated a new and objective method of 
evaluation to be applied to the JOS man-machine interface. Further evolution 
and systematic usage of this powerful tool will enable development of stan- 
dards and criteria for IOS design and evaluation that will assist the de- 
signer to arrive at an optimal system configuration and layout, quickly and 
accurately. 
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ABBREVIATIONS AND TERMS 


By - AKB 


CAM 

CRT 

ergonomic 

10S 

10S Activity List 


IPTASK 


ITD 


ITL 


jet trainer 


LVDU 
RTP 


RYDU 
SAINT 


TAM 


VOU 


auxiliary keyboard 

crew actions monitor 

cathode ray tube 

of or pertaining to biotechnology 
instructor/operator station 


element tasks as related to the use of a given 
device or function 


information presentation task: the task dés- 
cribing, in quantitative terms, the display 
characteristics of a given device 


instructor task description: the analysis of 
typical tasks in terms of information exchanges 
at the I0S 


instructional task list: the list of instruc- 
tional tasks an instructor may be called on to 
perform 


a typical jet fighter simulator, the two I0S 
con*igurations of which were utilized for the 
purposes of this study 


left video display unit 


representative task profile: a set of typical 
training activities arranged to present tasks 
and workload levels similar to those of a train- 
ing session 


right video display unit 


systems analysis of integrated network of 
tasks: a modelling and simulation technique 
for analyzing a set of problems from the field 
of human engineering 


typical activity module: a description of a 
typical instructor/I0S activity in sufficient 
detail so that most task requirements may be 
matched to most [05 configuration details 


video display unit 
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Page 6: At botiom of page. delete-"(the entire (TL is:available in Appendia A)” 

Page 7: In-third complete paragraph. line di: delete "(Appendix c)" 

Page 8: At bottom of page. delete “(the entire RTP is available in-Appendiy B)” 

Page 9: At bottom of page. delete “(the entire ITD listing ix available in Appendiy ()” 
Page 10: Th second complete paragraph, line 4. delete “(also-ineluded in- Appendix ¢)" 


Page Tl: AUbottoi. of page. delete “(the entire TAM listing is available.in Appendix: F)* 
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Page 19: In second paragraph. fourth line, delete “(Appeidix H)” . 
Page 31: In fifth tine from top of page. delete These values are presented in Appendiy G” 
Page 36: In fourth paragraph. first line. delete “(Appendix B);* 
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Page 36: liv fourth paragraph, first line, delete “(Appendis .B)” 

Page 38: En last paragraph.-fourth line, deleté “(Appendix BE)" 

Page 48:.1n second complete paragraph, thirteenthlines delete “(Appendis:G)” 
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delete last sentence. “The remainder of the TAMs..and their flowgraphs. can be found in Appendis B.~ 
Page 59: Délete first complete paragraph *The.complete set of nodule flowgraphs. wong with the 
SAINT data foreach module. is presented in Appeadis D.~ 
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Page 68: In first complete paragraph, -fifth line from-bottom: delete “(Appendix.:G)." In last 
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Page 69: In fourth paragraph. second’ tine.-delete “(see User's Guide, Appendix 1)* 


Page 84: In first paragraph, last two lines: deleié “(sed User's. Guide. Appendis 1)” 
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Feferencing the task number with the ITD. A complete printout of the results of RTP leg 3. Configurations 
No, | and. No.2. can be.found ii Appendis-1.” 


Page'119: In second coniplete-paragraph, first and second lines, délete “(see Appetidisd for the full: 
printout)” 
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